2024-03-11T21:39:24.054 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T21:39:24.054 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591936 --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose --description orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591936/orig.config.yaml 2024-03-11T21:39:24.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:39:24.162 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591936 branch: squid description: orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591936' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPZHMQSYU+9xQBMx3Zo5nsYRrzCiAe7m2+1dZOW81JPoY1rQD46GAGTyh16ltdE1KQogfW4osr72kIeHZMIRQ7I= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNrkTA4yWO3APhBiB0rpwLMjmsdXXvPQ+98Q1m3QdN9ujgtQyWAobm2jfxlz5SlWIphwOTgvCky6qC8gDQjZfGw= tasks: - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T21:39:24.162 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T21:39:24.163 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T21:39:24.163 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T21:39:24.165 INFO:teuthology.task.internal:Checking packages... 2024-03-11T21:39:24.188 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T21:39:24.188 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T21:39:24.188 INFO:teuthology.packaging:ref: None 2024-03-11T21:39:24.188 INFO:teuthology.packaging:tag: None 2024-03-11T21:39:24.188 INFO:teuthology.packaging:branch: squid 2024-03-11T21:39:24.188 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:39:24.189 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T21:39:24.612 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T21:39:24.614 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T21:39:24.619 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T21:39:24.620 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T21:39:24.643 INFO:teuthology.task.internal:Saving configuration 2024-03-11T21:39:24.655 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T21:39:24.661 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:39:24.685 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591936', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:34:21.952301', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPZHMQSYU+9xQBMx3Zo5nsYRrzCiAe7m2+1dZOW81JPoY1rQD46GAGTyh16ltdE1KQogfW4osr72kIeHZMIRQ7I='} 2024-03-11T21:39:24.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591936', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:34:21.953603', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNrkTA4yWO3APhBiB0rpwLMjmsdXXvPQ+98Q1m3QdN9ujgtQyWAobm2jfxlz5SlWIphwOTgvCky6qC8gDQjZfGw='} 2024-03-11T21:39:24.706 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T21:39:24.711 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T21:39:24.711 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-11T21:39:24.711 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T21:39:24.784 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdf5ca29f70>, signals=[15]) 2024-03-11T21:39:24.784 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T21:39:24.791 INFO:teuthology.task.internal:Opening connections... 2024-03-11T21:39:24.792 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-03-11T21:39:24.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:39:24.888 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-03-11T21:39:24.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:39:24.978 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T21:39:24.985 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-03-11T21:39:25.001 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-03-11T21:39:25.002 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:39:25.058 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:39:25.059 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:39:25.059 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:39:25.059 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-03-11T21:39:25.083 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-03-11T21:39:25.101 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-03-11T21:39:25.101 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-03-11T21:39:25.158 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-03-11T21:39:25.158 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-03-11T21:39:25.158 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-03-11T21:39:25.158 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:39:25.158 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-03-11T21:39:25.158 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:39:25.159 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:39:25.159 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-03-11T21:39:25.159 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:39:25.159 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:39:25.159 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:39:25.159 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:39:25.159 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:39:25.159 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:39:25.159 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-03-11T21:39:25.182 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T21:39:25.190 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T21:39:25.195 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T21:39:25.196 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:39:25.198 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:39:25.214 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T21:39:25.220 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T21:39:25.220 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:39:25.254 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:39:25.385 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T21:39:25.439 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T21:39:25.456 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T21:39:25.456 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T21:39:25.456 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:39:25.457 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:39:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.473 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-11T21:39:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.474 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-03-11T21:39:25.528 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:39:25.529 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-361.el9.x86_64 2024-03-11T21:39:25.529 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-03-11T21:39:25.530 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:39:25.530 INFO:teuthology.task.kernel:Running kernel on smithi102: 5.14.0-361.el9.x86_64 2024-03-11T21:39:25.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2024-03-11T21:39:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-11T21:39:27.670 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-11T21:39:29.239 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-11T21:39:29.329 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-11T21:39:30.863 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.0 MB 00:01 2024-03-11T21:39:40.238 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-11T21:39:41.702 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 684 kB/s | 8.0 MB 00:12 2024-03-11T21:39:46.789 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-03-11T21:39:46.903 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 19 MB 00:04 2024-03-11T21:39:49.331 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-11T21:39:52.420 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.1 MB 00:02 2024-03-11T21:39:54.480 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 15 kB 00:00 2024-03-11T21:39:55.197 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 35 MB/s | 21 MB 00:00 2024-03-11T21:39:58.535 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 2.3 MB/s | 21 MB 00:09 2024-03-11T21:40:00.206 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-11T21:40:02.471 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:40:02.472 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:40:02.574 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-11T21:40:02.580 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-11T21:40:02.580 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-03-11T21:40:02.580 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-11T21:40:02.580 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-03-11T21:40:02.581 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:40:02.582 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 100 M 2024-03-11T21:40:02.582 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 123 M 2024-03-11T21:40:02.582 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-11T21:40:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-11T21:40:05.332 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 2.4 MB/s | 6.0 MB 00:02 2024-03-11T21:40:05.572 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:40:05.572 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:40:05.674 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-11T21:40:05.678 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-11T21:40:05.678 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout:Install 4 Packages 2024-03-11T21:40:05.679 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:40:05.680 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 100 M 2024-03-11T21:40:05.680 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 123 M 2024-03-11T21:40:05.680 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-11T21:40:07.192 INFO:teuthology.orchestra.run.smithi102.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.1 MB/s | 6.0 MB 00:01 2024-03-11T21:40:07.366 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 8.7 MB/s | 39 MB 00:04 2024-03-11T21:40:10.532 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 2.7 MB/s | 21 MB 00:07 2024-03-11T21:40:10.600 INFO:teuthology.orchestra.run.smithi102.stdout:(2/4): kernel-modules-core-5.14.0-427.el9.x86_6 9.8 MB/s | 33 MB 00:03 2024-03-11T21:40:12.274 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 4.8 MB/s | 33 MB 00:06 2024-03-11T21:40:12.277 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:40:12.277 INFO:teuthology.orchestra.run.smithi110.stdout:Total 10 MB/s | 100 MB 00:09 2024-03-11T21:40:13.157 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-11T21:40:13.519 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-11T21:40:13.519 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-11T21:40:15.172 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-11T21:40:15.172 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-11T21:40:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-11T21:40:18.812 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:40:18.935 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:40:21.732 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:40:22.066 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:40:26.795 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:40:26.990 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:40:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:40:38.576 INFO:teuthology.orchestra.run.smithi102.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 650 kB/s | 21 MB 00:32 2024-03-11T21:40:56.080 INFO:teuthology.orchestra.run.smithi102.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 800 kB/s | 39 MB 00:50 2024-03-11T21:40:56.081 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:40:56.081 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.0 MB/s | 100 MB 00:50 2024-03-11T21:40:56.960 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-11T21:40:57.314 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-11T21:40:57.314 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-11T21:40:58.947 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-11T21:40:58.948 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-11T21:41:00.520 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:00.520 INFO:teuthology.orchestra.run.smithi110.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:41:00.520 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:41:00.605 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:01.124 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:01.124 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:41:01.124 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:41:01.124 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:41:01.525 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:01.525 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:41:01.525 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-11T21:41:01.525 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:41:01.525 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:41:01.525 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:41:01.525 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:41:01.525 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:41:01.525 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-11T21:41:01.866 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-11T21:41:02.056 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:41:02.190 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:41:02.190 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:41:02.190 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:41:02.190 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:41:02.190 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:41:02.190 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T21:41:02.191 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:41:02.191 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:41:02.309 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:41:02.310 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:41:02.310 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2024-03-11T21:41:02.354 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.06 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 68.el9 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Group : Unspecified 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 8264254 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:41:02.355 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : builder@centos.org 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:41:02.356 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:41:02.357 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:41:02.357 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:41:02.357 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:41:02.512 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:41:02.640 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:41:02.938 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-03-11T21:41:04.305 INFO:teuthology.orchestra.run.smithi110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:41:04.348 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-03-11T21:41:04.350 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-03-11T21:41:04.367 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.MmRtcWW9s3 2024-03-11T21:41:04.367 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MmRtcWW9s3 2024-03-11T21:41:04.435 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.MmRtcWW9s3 2024-03-11T21:41:04.578 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.MmRtcWW9s3 is 6KB 2024-03-11T21:41:04.626 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.MmRtcWW9s3 2024-03-11T21:41:04.640 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:41:04.710 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:41:04.710 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:41:04.710 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:41:04.710 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:41:04.711 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:41:04.871 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-03-11T21:41:05.439 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:41:05.619 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:41:10.325 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:41:10.538 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:15.308 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:34.874 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T21:41:34.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-11T21:41:34.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:41:44.816 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:44.816 INFO:teuthology.orchestra.run.smithi102.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:41:44.816 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:41:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:45.402 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:45.403 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:41:45.403 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:41:45.403 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:41:45.830 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:41:45.830 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:41:45.830 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-11T21:41:45.830 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:41:45.830 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:41:45.830 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:41:45.830 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:41:45.830 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:41:45.830 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-11T21:41:46.354 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:41:46.489 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:41:46.490 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:41:46.490 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:41:46.490 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:41:46.490 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:41:46.490 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T21:41:46.490 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:41:46.490 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:41:46.610 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:41:46.610 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:41:46.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 68.el9 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8264254 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:41:46.649 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:41:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:41:46.651 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:41:46.651 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:41:46.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:41:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-03-11T21:41:48.603 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:41:48.629 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-03-11T21:41:48.632 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-11T21:41:48.647 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.rEUOGvizCi 2024-03-11T21:41:48.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rEUOGvizCi 2024-03-11T21:41:48.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.rEUOGvizCi 2024-03-11T21:41:48.934 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.rEUOGvizCi is 6KB 2024-03-11T21:41:48.984 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.rEUOGvizCi 2024-03-11T21:41:48.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:41:49.063 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:41:49.063 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:41:49.063 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:41:49.063 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:41:49.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:41:49.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-03-11T21:42:19.250 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T21:42:19.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-11T21:42:19.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:42:34.891 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T21:42:37.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-11T21:42:43.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-11T21:42:43.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:42:44.299 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-03-11T21:42:44.864 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-11T21:42:44.864 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:42:44.865 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-03-11T21:42:44.879 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:42:44.879 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:42:44.879 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:42:44.879 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:42:45.880 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T21:42:45.880 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:42:45.954 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-03-11T21:42:45.994 DEBUG:teuthology.parallel:result is None 2024-03-11T21:42:46.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-11T21:42:46.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:42:49.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-11T21:43:01.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-11T21:43:01.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:43:33.511 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-11T21:43:33.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-11T21:43:33.908 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:43:33.908 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-11T21:43:33.922 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:43:33.922 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:43:33.923 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:43:33.923 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:43:34.924 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T21:43:34.924 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:43:35.021 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-03-11T21:43:35.070 DEBUG:teuthology.parallel:result is None 2024-03-11T21:43:35.071 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T21:43:35.078 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T21:43:35.078 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:43:35.080 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:43:35.099 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T21:43:35.107 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T21:43:35.115 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T21:43:35.115 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:43:35.135 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:43:35.171 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T21:43:35.181 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T21:43:35.181 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:43:35.194 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:43:35.219 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:43:35.232 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:43:35.255 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:43:35.269 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:43:35.271 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T21:43:35.278 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T21:43:35.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:43:35.281 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:43:35.309 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T21:43:35.319 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T21:43:35.319 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:43:35.349 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:43:35.366 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:43:35.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:43:35.492 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:43:35.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:43:35.627 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:43:35.627 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:43:35.693 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:43:35.739 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:43:35.791 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:43:35.876 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:43:35.918 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:43:35.918 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:43:35.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-03-11T21:43:35.988 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-03-11T21:43:36.097 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:43:36.107 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:43:36.517 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T21:43:36.525 INFO:teuthology.task.internal:Starting timer... 2024-03-11T21:43:36.525 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T21:43:36.537 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T21:43:36.545 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T21:43:36.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-03-11T21:43:36.623 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-03-11T21:43:36.626 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-03-11T21:43:36.678 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-03-11T21:43:36.680 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T21:43:36.680 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-03-11T21:43:36.729 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-03-11T21:43:36.729 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-03-11T21:43:36.758 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-03-11T21:43:36.758 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi110.front.sepia.ceph.com': 'permissive'} 2024-03-11T21:43:36.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:43:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.818 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:43:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.847 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T21:43:36.847 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:43:36.990 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:43:37.026 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T21:43:37.035 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T21:43:37.053 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T21:43:37.054 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T21:47:45.433 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com')] 2024-03-11T21:47:45.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-11T21:47:45.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:47:45.525 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-11T21:47:45.600 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-11T21:47:45.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-11T21:47:45.601 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:47:45.686 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-03-11T21:47:45.765 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-11T21:47:45.765 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T21:47:45.775 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T21:47:45.775 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:47:45.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:47:45.778 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:47:45.778 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:47:45.806 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:47:45.822 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:47:45.836 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:47:45.851 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-03-11T21:47:45.852 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:47:45.865 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-03-11T21:47:45.881 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:47:45.881 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-03-11T21:47:45.895 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-03-11T21:47:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:47:45.912 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:47:45.928 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:47:45.946 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:47:45.949 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:47:45.949 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-11T21:47:45.950 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:47:45.950 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:47:45.950 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:47:45.950 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:47:45.983 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:47:45.987 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:47:45.987 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-03-11T21:47:45.987 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:47:45.987 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:47:45.987 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:47:45.987 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:47:45.987 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-11T21:47:45.997 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-11T21:47:45.997 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:47:45.997 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:47:46.012 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:47:46.012 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-03-11T21:47:46.067 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:47:46.067 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:47:46.067 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T21:47:46.067 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:47:46.067 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:47:46.067 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-11 21:47:44.511755532 +0000 2024-03-11T21:47:46.067 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-11 21:47:23.113193273 +0000 2024-03-11T21:47:46.067 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-11 21:47:23.113193273 +0000 2024-03-11T21:47:46.067 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-11 21:47:23.113193273 +0000 2024-03-11T21:47:46.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:47:46.132 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-11T21:47:46.132 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-11T21:47:46.133 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000196209 s, 2.6 MB/s 2024-03-11T21:47:46.134 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:47:46.189 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-03-11T21:47:46.243 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:47:46.243 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:47:46.243 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T21:47:46.243 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:47:46.243 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:47:46.243 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-11 21:47:44.511755532 +0000 2024-03-11T21:47:46.243 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-11 21:47:23.436186666 +0000 2024-03-11T21:47:46.243 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-11 21:47:23.436186666 +0000 2024-03-11T21:47:46.243 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-11 21:47:23.436186666 +0000 2024-03-11T21:47:46.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:47:46.308 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-11T21:47:46.308 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-11T21:47:46.308 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000158763 s, 3.2 MB/s 2024-03-11T21:47:46.309 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:47:46.364 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-03-11T21:47:46.419 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:47:46.419 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:47:46.419 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-11T21:47:46.419 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:47:46.419 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:47:46.419 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-11 21:47:44.511755532 +0000 2024-03-11T21:47:46.419 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-11 21:47:23.844178320 +0000 2024-03-11T21:47:46.419 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-11 21:47:23.844178320 +0000 2024-03-11T21:47:46.419 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-11 21:47:23.844178320 +0000 2024-03-11T21:47:46.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:47:46.484 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-11T21:47:46.484 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-11T21:47:46.484 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000208138 s, 2.5 MB/s 2024-03-11T21:47:46.485 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:47:46.541 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-03-11T21:47:46.596 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:47:46.597 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:47:46.597 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T21:47:46.597 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:47:46.597 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:47:46.597 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-11 21:47:44.511755532 +0000 2024-03-11T21:47:46.597 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-11 21:47:24.186171324 +0000 2024-03-11T21:47:46.597 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-11 21:47:24.186171324 +0000 2024-03-11T21:47:46.597 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-11 21:47:24.186171324 +0000 2024-03-11T21:47:46.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:47:46.662 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-11T21:47:46.662 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-11T21:47:46.662 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000210345 s, 2.4 MB/s 2024-03-11T21:47:46.663 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:47:46.718 DEBUG:teuthology.orchestra.run.smithi102:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:47:46.862 INFO:teuthology.orchestra.run.smithi102.stdout:loop 2024-03-11T21:47:46.863 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_1... 2024-03-11T21:47:46.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:47:46.901 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-11T21:47:46.938 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:47:46.968 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_2... 2024-03-11T21:47:46.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:47:47.046 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-11T21:47:47.084 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:47:47.112 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_3... 2024-03-11T21:47:47.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:47:47.314 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-11T21:47:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:47:47.390 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_4... 2024-03-11T21:47:47.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:47:47.469 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-11T21:47:47.506 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:47:47.541 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:47:47.541 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:47:47.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme list 2024-03-11T21:47:47.659 INFO:teuthology.orchestra.run.smithi102.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:47:47.659 INFO:teuthology.orchestra.run.smithi102.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:47:47.660 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme0n1 PHFT620400VK400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T21:47:47.660 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme1n1 fe33302851940c7a99cb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:47:47.660 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme2n1 e418c6793f1896d0b0d3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:47:47.660 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme3n1 c4bc35a5e15294c2d117 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:47:47.660 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme4n1 2008554550ed85afad45 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:47:47.661 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:47:47.661 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:47:47.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/scratch_devs 2024-03-11T21:47:47.727 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:47:47.728 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:47:47.747 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:47:47.747 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-03-11T21:47:47.804 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:47:47.804 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:47:47.804 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-11T21:47:47.804 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:47:47.804 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:47:47.805 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-11 21:47:44.481812778 +0000 2024-03-11T21:47:47.805 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-11 21:47:23.093202170 +0000 2024-03-11T21:47:47.805 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-11 21:47:23.093202170 +0000 2024-03-11T21:47:47.805 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-11 21:47:23.093202170 +0000 2024-03-11T21:47:47.805 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:47:47.875 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-11T21:47:47.875 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-11T21:47:47.875 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000116179 s, 4.4 MB/s 2024-03-11T21:47:47.876 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:47:47.935 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-03-11T21:47:47.992 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:47:47.992 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:47:47.992 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-11T21:47:47.992 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:47:47.992 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:47:47.992 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-11 21:47:44.481812778 +0000 2024-03-11T21:47:47.992 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-11 21:47:23.434195962 +0000 2024-03-11T21:47:47.992 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-11 21:47:23.434195962 +0000 2024-03-11T21:47:47.992 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-11 21:47:23.434195962 +0000 2024-03-11T21:47:47.992 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:47:48.059 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-11T21:47:48.059 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-11T21:47:48.059 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.00010609 s, 4.8 MB/s 2024-03-11T21:47:48.060 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:47:48.117 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-03-11T21:47:48.173 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:47:48.173 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:47:48.173 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-11T21:47:48.173 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:47:48.173 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:47:48.173 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-11 21:47:44.481812778 +0000 2024-03-11T21:47:48.173 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-11 21:47:23.748190246 +0000 2024-03-11T21:47:48.174 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-11 21:47:23.748190246 +0000 2024-03-11T21:47:48.174 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-11 21:47:23.748190246 +0000 2024-03-11T21:47:48.174 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:47:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-11T21:47:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-11T21:47:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.5922e-05 s, 6.0 MB/s 2024-03-11T21:47:48.241 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:47:48.296 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-03-11T21:47:48.351 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:47:48.351 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:47:48.351 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T21:47:48.351 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:47:48.351 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:47:48.351 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-11 21:47:44.481812778 +0000 2024-03-11T21:47:48.351 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-11 21:47:24.074184311 +0000 2024-03-11T21:47:48.351 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-11 21:47:24.074184311 +0000 2024-03-11T21:47:48.352 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-11 21:47:24.074184311 +0000 2024-03-11T21:47:48.352 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:47:48.416 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-11T21:47:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-11T21:47:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.0962e-05 s, 6.3 MB/s 2024-03-11T21:47:48.418 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:47:48.473 DEBUG:teuthology.orchestra.run.smithi110:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:47:48.617 INFO:teuthology.orchestra.run.smithi110.stdout:loop 2024-03-11T21:47:48.618 INFO:tasks.nvme_loop:Connecting nvme_loop smithi110:/dev/vg_nvme/lv_1... 2024-03-11T21:47:48.618 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:47:48.655 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-03-11T21:47:48.693 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:47:48.722 INFO:tasks.nvme_loop:Connecting nvme_loop smithi110:/dev/vg_nvme/lv_2... 2024-03-11T21:47:48.722 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:47:48.800 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-03-11T21:47:48.837 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:47:48.865 INFO:tasks.nvme_loop:Connecting nvme_loop smithi110:/dev/vg_nvme/lv_3... 2024-03-11T21:47:48.865 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:47:48.943 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-03-11T21:47:48.982 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:47:49.008 INFO:tasks.nvme_loop:Connecting nvme_loop smithi110:/dev/vg_nvme/lv_4... 2024-03-11T21:47:49.009 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:47:49.086 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-03-11T21:47:49.123 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:47:49.163 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:47:49.163 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:47:49.219 DEBUG:teuthology.orchestra.run.smithi110:> sudo nvme list 2024-03-11T21:47:49.284 INFO:teuthology.orchestra.run.smithi110.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:47:49.284 INFO:teuthology.orchestra.run.smithi110.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:47:49.284 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme0n1 PHMB7476002T480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-11T21:47:49.284 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme1n1 57ede36057af39640ba0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:47:49.284 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme2n1 8dd66951b051d85857b8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:47:49.284 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme3n1 0f094762ce4cede9c919 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:47:49.284 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme4n1 8160aeb7118ce9398ce1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:47:49.285 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:47:49.285 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:47:49.285 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/scratch_devs 2024-03-11T21:47:49.352 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T21:47:49.449 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T21:47:49.450 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:47:49.450 INFO:tasks.cephadm:Cluster fsid is 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:47:49.450 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T21:47:49.450 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.102', 'mon.c': '[v2:172.21.15.102:3301,v1:172.21.15.102:6790]', 'mon.b': '172.21.15.110'} 2024-03-11T21:47:49.450 INFO:tasks.cephadm:First mon is mon.a on smithi102 2024-03-11T21:47:49.450 INFO:tasks.cephadm:First mgr is y 2024-03-11T21:47:49.451 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T21:47:49.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2024-03-11T21:47:49.482 DEBUG:teuthology.orchestra.run.smithi110:> sudo hostname $(hostname -s) 2024-03-11T21:47:49.509 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T21:47:49.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:47:49.736 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T21:47:49.877 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:47:49.879 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T21:47:49.879 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:47:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 21:47 /home/ubuntu/cephtest/cephadm 2024-03-11T21:47:50.330 DEBUG:teuthology.orchestra.run.smithi110:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:47:50.918 INFO:teuthology.orchestra.run.smithi110.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 21:47 /home/ubuntu/cephtest/cephadm 2024-03-11T21:47:50.918 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:47:50.936 DEBUG:teuthology.orchestra.run.smithi110:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:47:50.964 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T21:47:50.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T21:47:50.978 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T21:47:51.719 INFO:teuthology.orchestra.run.smithi110.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:47:51.872 INFO:teuthology.orchestra.run.smithi102.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:49:01.146 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-11T21:49:01.147 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T21:49:01.147 INFO:teuthology.orchestra.run.smithi102.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T21:49:01.147 INFO:teuthology.orchestra.run.smithi102.stdout: "repo_digests": [ 2024-03-11T21:49:01.147 INFO:teuthology.orchestra.run.smithi102.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T21:49:01.147 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-11T21:49:01.147 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-11T21:49:06.361 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-03-11T21:49:06.361 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T21:49:06.361 INFO:teuthology.orchestra.run.smithi110.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T21:49:06.361 INFO:teuthology.orchestra.run.smithi110.stdout: "repo_digests": [ 2024-03-11T21:49:06.361 INFO:teuthology.orchestra.run.smithi110.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T21:49:06.361 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-03-11T21:49:06.361 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-03-11T21:49:06.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2024-03-11T21:49:06.422 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph 2024-03-11T21:49:06.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2024-03-11T21:49:06.494 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /etc/ceph 2024-03-11T21:49:06.519 INFO:tasks.cephadm:Writing seed config... 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T21:49:06.521 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T21:49:06.522 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:49:06.522 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T21:49:06.551 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0126e6b2-dff1-11ee-95c8-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-11T21:49:06.552 DEBUG:teuthology.orchestra.run.smithi102:mon.a> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T21:49:06.594 DEBUG:teuthology.orchestra.run.smithi102:mgr.y> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T21:49:06.637 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T21:49:06.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.102 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:49:06.845 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:49:06.845 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', '0126e6b2-dff1-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.102', '--skip-admin-label'] 2024-03-11T21:49:06.871 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:49:06.871 INFO:teuthology.orchestra.run.smithi102.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T21:49:06.872 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying podman|docker is present... 2024-03-11T21:49:06.896 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:49:06.896 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying lvm2 is present... 2024-03-11T21:49:06.896 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying time synchronization is in place... 2024-03-11T21:49:06.903 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:49:06.903 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:49:06.908 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:49:06.909 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-11T21:49:06.915 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-03-11T21:49:06.920 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-03-11T21:49:06.921 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:49:06.921 INFO:teuthology.orchestra.run.smithi102.stdout:Repeating the final host check... 2024-03-11T21:49:06.943 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:49:06.943 INFO:teuthology.orchestra.run.smithi102.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T21:49:06.943 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl is present 2024-03-11T21:49:06.943 INFO:teuthology.orchestra.run.smithi102.stdout:lvcreate is present 2024-03-11T21:49:06.949 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:49:06.949 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:49:06.955 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:49:06.955 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-11T21:49:06.961 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-03-11T21:49:06.967 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-03-11T21:49:06.967 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:49:06.968 INFO:teuthology.orchestra.run.smithi102.stdout:Host looks OK 2024-03-11T21:49:06.968 INFO:teuthology.orchestra.run.smithi102.stdout:Cluster fsid: 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:06.968 INFO:teuthology.orchestra.run.smithi102.stdout:Acquiring lock 139630318150704 on /run/cephadm/0126e6b2-dff1-11ee-95c8-87774f69a715.lock 2024-03-11T21:49:06.968 INFO:teuthology.orchestra.run.smithi102.stdout:Lock 139630318150704 acquired on /run/cephadm/0126e6b2-dff1-11ee-95c8-87774f69a715.lock 2024-03-11T21:49:06.969 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 3300 ... 2024-03-11T21:49:06.969 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 6789 ... 2024-03-11T21:49:06.969 INFO:teuthology.orchestra.run.smithi102.stdout:Base mon IP(s) is [172.21.15.102:3300, 172.21.15.102:6789], mon addrv is [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-03-11T21:49:06.971 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.102 metric 100 2024-03-11T21:49:06.972 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.102 metric 100 2024-03-11T21:49:06.974 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T21:49:06.974 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T21:49:06.974 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-03-11T21:49:06.976 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T21:49:06.977 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T21:49:06.977 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:49:06.977 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T21:49:06.977 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-11T21:49:06.977 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:49:06.977 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-03-11T21:49:06.978 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-03-11T21:49:06.978 INFO:teuthology.orchestra.run.smithi102.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T21:49:06.978 INFO:teuthology.orchestra.run.smithi102.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T21:49:06.978 INFO:teuthology.orchestra.run.smithi102.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:49:07.439 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T21:49:07.440 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:49:07.440 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T21:49:07.440 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T21:49:07.440 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T21:49:07.440 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T21:49:07.440 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T21:49:08.378 INFO:teuthology.orchestra.run.smithi102.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T21:49:08.378 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T21:49:08.378 INFO:teuthology.orchestra.run.smithi102.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T21:49:09.193 INFO:teuthology.orchestra.run.smithi102.stdout:stat: stdout 167 167 2024-03-11T21:49:09.193 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial keys... 2024-03-11T21:49:10.014 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQBVfO9lTPDRHRAANVVszsfcs5HGKU8E1auQEg== 2024-03-11T21:49:10.844 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQBWfO9ltd44ERAA0+fCmIFEdjj+FFyzp5Pz8w== 2024-03-11T21:49:11.649 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQBXfO9lipyZCBAAoBzCFI6wO4KdWMHvZEZ7CA== 2024-03-11T21:49:11.650 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial monmap... 2024-03-11T21:49:12.514 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:49:12.514 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T21:49:12.514 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:12.515 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:49:12.515 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool for a [v2:172.21.15.102:3300,v1:172.21.15.102:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:49:12.515 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = quincy 2024-03-11T21:49:12.515 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: set fsid to 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:12.515 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:49:12.515 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:49:12.515 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mon... 2024-03-11T21:49:13.370 INFO:teuthology.orchestra.run.smithi102.stdout:create mon.a on 2024-03-11T21:49:13.718 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T21:49:13.894 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715.target. 2024-03-11T21:49:13.894 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715.target. 2024-03-11T21:49:14.129 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:49:14.129 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T21:49:14.303 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715.target.wants/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@.service. 2024-03-11T21:49:14.820 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:49:14.820 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-11T21:49:14.827 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:49:14.827 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-11T21:49:14.827 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-11T21:49:14.827 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:49:14.828 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon to start... 2024-03-11T21:49:14.828 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon... 2024-03-11T21:49:15.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T21:49:15.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout id: 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:15.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T21:49:15.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:15.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout services: 2024-03-11T21:49:15.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.484739s) 2024-03-11T21:49:15.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T21:49:15.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T21:49:15.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:15.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout data: 2024-03-11T21:49:15.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T21:49:15.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T21:49:15.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T21:49:15.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T21:49:15.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:15.817 INFO:teuthology.orchestra.run.smithi102.stdout:mon is available 2024-03-11T21:49:15.817 INFO:teuthology.orchestra.run.smithi102.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T21:49:15.904 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:15 smithi102 ceph-mon[24961]: from='client.? 172.21.15.102:0/4159295' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T21:49:17.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:16 smithi102 ceph-mon[24961]: from='client.? 172.21.15.102:0/1681999989' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:49:17.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:16 smithi102 ceph-mon[24961]: from='client.? 172.21.15.102:0/1681999989' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T21:49:21.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:21.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:49:21.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:21.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-03-11T21:49:21.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:49:21.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:49:21.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:49:21.125 INFO:teuthology.orchestra.run.smithi102.stdout:Generating new minimal ceph.conf... 2024-03-11T21:49:24.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:24 smithi102 ceph-mon[24961]: from='client.? 172.21.15.102:0/2065762438' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:24.895 INFO:teuthology.orchestra.run.smithi102.stdout:Restarting the monitor... 2024-03-11T21:49:24.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:24 smithi102 systemd[1]: Stopping Ceph mon.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:49:25.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:25 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a[24955]: 2024-03-11T21:49:25.145+0000 7fc805a46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:49:25.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:25 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a[24955]: 2024-03-11T21:49:25.145+0000 7fc805a46700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T21:49:25.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:25 smithi102 podman[25207]: 2024-03-11 21:49:25.401828934 +0000 UTC m=+0.338358816 container died 4eee41ce42d84976cc88c9ad7ee7e68395f76c1fd78fc40c8813fd30fa689207 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-11T21:49:26.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:25 smithi102 podman[25207]: 2024-03-11 21:49:25.703674802 +0000 UTC m=+0.640204689 container cleanup 4eee41ce42d84976cc88c9ad7ee7e68395f76c1fd78fc40c8813fd30fa689207 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T21:49:26.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:25 smithi102 bash[25207]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:49:26.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:25 smithi102 podman[25219]: 2024-03-11 21:49:25.91294432 +0000 UTC m=+0.507054676 container remove 4eee41ce42d84976cc88c9ad7ee7e68395f76c1fd78fc40c8813fd30fa689207 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:49:26.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T21:49:26.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 systemd[1]: Stopped Ceph mon.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:49:26.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 systemd[1]: Starting Ceph mon.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:49:26.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 podman[25319]: 2024-03-11 21:49:26.238643405 +0000 UTC m=+0.093064142 container create e26e085d973f9870f4d1f248cfadf9e5bb30557be480efd2e462801928471879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T21:49:26.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 podman[25319]: 2024-03-11 21:49:26.164458966 +0000 UTC m=+0.018879713 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:49:26.511 INFO:teuthology.orchestra.run.smithi102.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T21:49:26.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 podman[25319]: 2024-03-11 21:49:26.4498238 +0000 UTC m=+0.304244537 container init e26e085d973f9870f4d1f248cfadf9e5bb30557be480efd2e462801928471879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-11T21:49:26.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 podman[25319]: 2024-03-11 21:49:26.452720825 +0000 UTC m=+0.307141563 container start e26e085d973f9870f4d1f248cfadf9e5bb30557be480efd2e462801928471879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-11T21:49:26.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:49:26.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T21:49:26.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: pidfile_write: ignore empty --pid-file 2024-03-11T21:49:26.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: load: jerasure load: lrc 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Git sha 0 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: DB SUMMARY 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: DB Session ID: E7I5U54Q99QF2I28QW6G 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84645 ; 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:49:26.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.env: 0x5559baca5ae0 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.info_log: 0x5559bc1f98c0 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.statistics: (nil) 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.use_fsync: 0 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:49:26.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.db_log_dir: 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.wal_dir: 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:49:26.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.write_buffer_manager: 0x5559bc1cd400 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.unordered_write: 0 2024-03-11T21:49:26.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:49:26.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:49:26.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:49:26.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:49:26.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.row_cache: None 2024-03-11T21:49:26.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.wal_filter: None 2024-03-11T21:49:26.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:49:26.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:49:26.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:49:26.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:49:26.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.wal_compression: 0 2024-03-11T21:49:26.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:49:26.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:49:26.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_open_files: -1 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:49:26.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Compression algorithms supported: 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: kZSTD supported: 0 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:49:26.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.merge_operator: 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_filter: None 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5559bc1f99c0) 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_type: 0 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_index_type: 0 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_shortening: 1 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:49:26.636 INFO:journalctl@ceph.mon.a.smithi102.stdout: checksum: 4 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: no_block_cache: 0 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache: 0x5559bc1fd090 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_options: 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: capacity : 536870912 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_shard_bits : 4 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: strict_capacity_limit : 0 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_compressed: (nil) 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: persistent_cache: (nil) 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size: 4096 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size_deviation: 10 2024-03-11T21:49:26.637 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_restart_interval: 16 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_block_restart_interval: 1 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: metadata_block_size: 4096 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: partition_filters: 0 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: use_delta_encoding: 1 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: filter_policy: bloomfilter 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: whole_key_filtering: 1 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: verify_compression: 0 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: format_version: 5 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_index_compression: 1 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_align: 0 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_auto_readahead_size: 262144 2024-03-11T21:49:26.638 INFO:journalctl@ceph.mon.a.smithi102.stdout: prepopulate_block_cache: 0 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression: NoCompression 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.num_levels: 7 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:49:26.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:49:26.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:49:26.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.ttl: 2592000 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:49:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 21b8a318-970d-4b61-89a2-04ce6fbbbcb3 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193766481185, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T21:49:26.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193766482027, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81270, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79457, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73766, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710193766, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "21b8a318-970d-4b61-89a2-04ce6fbbbcb3", "db_session_id": "E7I5U54Q99QF2I28QW6G", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193766482106, "job": 1, "event": "recovery_finished"} 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5559bc2ee000 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: DB pointer 0x5559bc2d8000 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** DB Stats ** 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:49:26.645 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: L0 2/0 81.20 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: Sum 2/0 81.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-11T21:49:26.646 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative compaction: 0.00 GB write, 27.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval compaction: 0.00 GB write, 27.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache BinnedLRUCache@0x5559bc1fd090#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.7e-05 secs_since: 0 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: starting mon.a rank 0 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:26.647 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mon.a@-1(???) e1 preinit fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mon.a@-1(???).mds e1 new map 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mon.a@-1(???).mds e1 print_map 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout: e1 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout: legacy client fscid: -1 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout: No filesystems configured 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 bash[25319]: e26e085d973f9870f4d1f248cfadf9e5bb30557be480efd2e462801928471879 2024-03-11T21:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 systemd[1]: Started Ceph mon.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:49:26.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:49:26.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: monmap e1: 1 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:49:26.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: fsmap 2024-03-11T21:49:26.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T21:49:26.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:26 smithi102 ceph-mon[25338]: mgrmap e1: no daemons active 2024-03-11T21:49:27.500 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T21:49:27.522 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:49:27.522 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mgr... 2024-03-11T21:49:27.522 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T21:49:27.523 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T21:49:27.760 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:49:27.761 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T21:49:27.906 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715.target.wants/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@.service. 2024-03-11T21:49:28.204 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:28 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2432561780' entity='client.admin' 2024-03-11T21:49:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:49:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-11T21:49:28.397 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:49:28.397 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-11T21:49:28.397 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-11T21:49:28.397 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:49:28.405 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:49:28.405 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-11T21:49:28.412 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:49:28.413 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-11T21:49:28.413 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-11T21:49:28.413 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T21:49:28.413 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr to start... 2024-03-11T21:49:28.413 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr... 2024-03-11T21:49:29.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:29 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/941018120' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:49:29.917 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:29 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:29.846+0000 7fe9b3f5c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:49:30.275 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:29 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:29.942+0000 7fe9b3f5c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:49:30.275 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:30 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:30.024+0000 7fe9b3f5c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:49:30.275 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:30 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:30.282+0000 7fe9b3f5c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:49:30.561 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:30 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:30.389+0000 7fe9b3f5c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:30.561 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:30 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:30.474+0000 7fe9b3f5c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:49:30.562 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:30 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:30.569+0000 7fe9b3f5c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:49:30.917 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:30 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:30.743+0000 7fe9b3f5c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:49:31.255 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:31.255 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:49:31.255 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "0126e6b2-dff1-11ee-95c8-87774f69a715", 2024-03-11T21:49:31.255 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:49:31.255 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:49:31.255 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:49:31.255 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:49:31.255 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:31.255 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:49:31.256 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:49:31.257 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:49:31.257 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:49:31.257 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:49:31.257 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:49:31.260 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:31.260 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:49:31.260 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:49:31.260 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:49:31.260 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:49:31.260 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:49:31.261 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:49:14.830790+0000", 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:49:31.262 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (1/15)... 2024-03-11T21:49:32.167 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:31 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:31.713+0000 7fe9b3f5c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:49:32.167 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:31 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:31.844+0000 7fe9b3f5c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:49:32.453 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:32.175+0000 7fe9b3f5c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:49:32.453 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:32.265+0000 7fe9b3f5c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:49:32.453 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:32.460+0000 7fe9b3f5c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:32.916 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:32.641+0000 7fe9b3f5c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:49:33.167 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:32.924+0000 7fe9b3f5c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:33.467 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:33 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:33.180+0000 7fe9b3f5c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:33.467 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:33 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:33.428+0000 7fe9b3f5c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:49:34.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:33 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2818523462' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:49:34.311 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:34.312 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:49:34.312 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "0126e6b2-dff1-11ee-95c8-87774f69a715", 2024-03-11T21:49:34.312 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:49:34.312 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:49:34.312 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:49:34.312 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:34.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:49:34.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:49:34.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:49:14.830790+0000", 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:49:34.316 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (2/15)... 2024-03-11T21:49:34.667 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:34 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:34.240+0000 7fe9b3f5c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:49:35.117 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:34 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:34.811+0000 7fe9b3f5c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:49:35.117 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:34 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:34.909+0000 7fe9b3f5c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:49:35.117 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:35.126+0000 7fe9b3f5c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:49:35.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: Activating manager daemon y 2024-03-11T21:49:35.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: mgrmap e2: y(active, starting, since 0.0029245s) 2024-03-11T21:49:35.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:49:35.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:49:35.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:49:35.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:35.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:49:35.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: Manager daemon y is now available 2024-03-11T21:49:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:49:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:49:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' 2024-03-11T21:49:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' 2024-03-11T21:49:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:35 smithi102 ceph-mon[25338]: from='mgr.14100 172.21.15.102:0/1613168428' entity='mgr.y' 2024-03-11T21:49:37.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:37 smithi102 ceph-mon[25338]: mgrmap e3: y(active, since 1.00656s) 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "0126e6b2-dff1-11ee-95c8-87774f69a715", 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:49:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:49:37.893 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:49:37.897 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:49:37.897 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:49:37.897 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:37.897 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:49:37.897 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:49:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:49:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:49:14.830790+0000", 2024-03-11T21:49:37.900 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:49:37.900 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:49:37.900 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:49:37.900 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:49:37.900 INFO:teuthology.orchestra.run.smithi102.stdout:mgr is available 2024-03-11T21:49:38.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:38 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2647564476' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:49:39.030 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:39.030 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:49:39.030 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:39.030 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:49:39.030 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:49:39.030 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:49:39.031 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:49:39.031 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:39.031 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:49:39.031 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:49:39.031 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-11T21:49:39.031 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:49:39.031 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:49:39.031 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:49:39.031 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling cephadm module... 2024-03-11T21:49:39.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:39 smithi102 ceph-mon[25338]: mgrmap e4: y(active, since 3s) 2024-03-11T21:49:39.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:39 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2533461303' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:49:39.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:39 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2533461303' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T21:49:40.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:40 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/3150974849' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T21:49:40.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:40 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ignoring --setuser ceph since I am not root 2024-03-11T21:49:40.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:40 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ignoring --setgroup ceph since I am not root 2024-03-11T21:49:40.774 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:40 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:40.715+0000 7f2626ae2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:49:41.062 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:40 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:40.810+0000 7f2626ae2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:49:41.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:41 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/3150974849' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T21:49:41.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:41 smithi102 ceph-mon[25338]: mgrmap e5: y(active, since 5s) 2024-03-11T21:49:41.590 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:41 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:41.461+0000 7f2626ae2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:49:41.591 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:41 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:41.560+0000 7f2626ae2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:49:41.840 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:41 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:41.702+0000 7f2626ae2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:49:41.841 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:41 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:41.799+0000 7f2626ae2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:49:41.844 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:49:41.844 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T21:49:41.844 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:49:41.844 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:49:41.844 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:49:41.844 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:49:41.844 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-03-11T21:49:41.844 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 5... 2024-03-11T21:49:42.118 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:41 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:41.888+0000 7f2626ae2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:49:42.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:42 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2616629290' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:49:42.417 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:42 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:42.159+0000 7f2626ae2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:49:42.417 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:42 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:42.272+0000 7f2626ae2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:42.417 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:42 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:42.357+0000 7f2626ae2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:49:42.917 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:42 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:42.452+0000 7f2626ae2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:49:42.917 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:42 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:42.626+0000 7f2626ae2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:49:43.917 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:43 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:43.581+0000 7f2626ae2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:49:43.917 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:43 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:43.711+0000 7f2626ae2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:49:44.319 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:44 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:44.042+0000 7f2626ae2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:49:44.319 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:44 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:44.132+0000 7f2626ae2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:49:44.319 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:44 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:44.327+0000 7f2626ae2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:44.667 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:44 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:44.511+0000 7f2626ae2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:49:45.043 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:44 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:44.794+0000 7f2626ae2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:45.043 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:45.051+0000 7f2626ae2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:45.417 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:45.296+0000 7f2626ae2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:49:46.417 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:46 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:46.121+0000 7f2626ae2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:49:46.993 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:46 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:46.692+0000 7f2626ae2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:49:46.994 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:46 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:46.789+0000 7f2626ae2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:49:46.994 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:49:47.002+0000 7f2626ae2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:49:47.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: Active manager daemon y restarted 2024-03-11T21:49:47.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: Activating manager daemon y 2024-03-11T21:49:47.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T21:49:47.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: mgrmap e6: y(active, starting, since 0.00300959s) 2024-03-11T21:49:47.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:47.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:49:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:49:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:49:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:49:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: Manager daemon y is now available 2024-03-11T21:49:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:49:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:49:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:47 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:48.288 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:48 smithi102 ceph-mon[25338]: Found migration_current of "None". Setting to last migration. 2024-03-11T21:49:48.288 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:48 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:48.288 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:48 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:49:48.289 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:48 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:49:48.289 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:48 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:49:48.289 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:48 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:49:48.289 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:48 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:48.289 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:48 smithi102 ceph-mon[25338]: mgrmap e7: y(active, since 1.00537s) 2024-03-11T21:49:48.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:49:48.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T21:49:48.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:49:48.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:49:48.542 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 5 is available 2024-03-11T21:49:48.542 INFO:teuthology.orchestra.run.smithi102.stdout:Setting orchestrator backend to cephadm... 2024-03-11T21:49:49.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:49 smithi102 ceph-mon[25338]: [11/Mar/2024:21:49:47] ENGINE Bus STARTING 2024-03-11T21:49:49.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:49 smithi102 ceph-mon[25338]: [11/Mar/2024:21:49:47] ENGINE Serving on https://172.21.15.102:7150 2024-03-11T21:49:49.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:49 smithi102 ceph-mon[25338]: [11/Mar/2024:21:49:47] ENGINE Serving on http://172.21.15.102:8765 2024-03-11T21:49:49.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:49 smithi102 ceph-mon[25338]: [11/Mar/2024:21:49:47] ENGINE Bus STARTED 2024-03-11T21:49:49.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:49 smithi102 ceph-mon[25338]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:49:49.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:49 smithi102 ceph-mon[25338]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:49:50.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:50 smithi102 ceph-mon[25338]: mgrmap e8: y(active, since 2s) 2024-03-11T21:49:50.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:50 smithi102 ceph-mon[25338]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:50.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:50 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:49:50.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:50 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:51.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T21:49:51.037 INFO:teuthology.orchestra.run.smithi102.stdout:Generating ssh key... 2024-03-11T21:49:51.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:51 smithi102 ceph-mon[25338]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:52.398 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: Generating public/private rsa key pair. 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: Your identification has been saved in /tmp/tmp1d2s9433/key. 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: Your public key has been saved in /tmp/tmp1d2s9433/key.pub. 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: The key fingerprint is: 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: SHA256:AG82oIye9NLlENKdEj8O1q6KOSkwSmtO5HtwwT39uxQ ceph-0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: The key's randomart image is: 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: +---[RSA 3072]----+ 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: | ..o= . | 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: | o.++* | 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: |..++o=B | 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: |o =oB=.+ | 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: | = o.+. SE | 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: |=o... .. | 2024-03-11T21:49:52.399 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: |+=+. .. | 2024-03-11T21:49:52.400 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: |B+o. .. | 2024-03-11T21:49:52.400 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: |*=. .. | 2024-03-11T21:49:52.400 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:49:52 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: +----[SHA256]-----+ 2024-03-11T21:49:53.260 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:53 smithi102 ceph-mon[25338]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:53.261 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:53 smithi102 ceph-mon[25338]: Generating ssh key... 2024-03-11T21:49:53.261 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:53 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:49:53.261 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:53 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:49:53.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6N/1i3OzSfShdQTUqm060yVIshIC9ep8XtunD2AMZ8N//M51PII1JqynSvbsp0Is3+lGUM6wiP60MYprP/Lb9mxz3XCP/28DN7cC1ymF2s8wkZuLGZOnRw+xF5+AFe6mN3GOMii27NN5EMfd0mtEZ1cTz5vi9+3pkWi+d60ZC+EIj1vdWl2Bun3OW2AWZHY1UbD1Oi3arGroTkbf6xYwaZ3+9K0fvKnyEGh9xbFXJaTN8wupPyV0YnT03Aj/qwKbcuPbRTLFM0hLpHLwqac91tPVbzgGpOLZZwqFHikKl93mP/Lt9dKEEcOkAnsa7TFvsz4AkIc/q2yOeS7bDxzCpeqcV44C4Psbq6iLKClrS/bfBtdX3yOB1smbbS11q9kjfJKdVl5H8bbbnd+s144gcoDkIMyDSHGZ8N+A9jHaQhQawHni3iJ/LXWtRXouBoHm+Oz/j0UkedzPCmDCpHyy3zMfNQT1K17Qu7yfR8SEYbx9pCkacSttYdwmHIiWULRE= ceph-0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:49:53.824 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:49:53.824 INFO:teuthology.orchestra.run.smithi102.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T21:49:53.825 INFO:teuthology.orchestra.run.smithi102.stdout:Adding host smithi102... 2024-03-11T21:49:54.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:54 smithi102 ceph-mon[25338]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:55.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:55 smithi102 ceph-mon[25338]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:55.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:55 smithi102 ceph-mon[25338]: Deploying cephadm binary to smithi102 2024-03-11T21:49:59.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Added host 'smithi102' with addr '172.21.15.102' 2024-03-11T21:49:59.274 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mon service... 2024-03-11T21:49:59.897 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:59 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:49:59.897 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:59 smithi102 ceph-mon[25338]: Added host smithi102 2024-03-11T21:49:59.897 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:49:59 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:00.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T21:50:00.413 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mgr service... 2024-03-11T21:50:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:00 smithi102 ceph-mon[25338]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:00 smithi102 ceph-mon[25338]: Saving service mon spec with placement count:5 2024-03-11T21:50:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:00 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:50:01.510 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T21:50:02.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:02 smithi102 ceph-mon[25338]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:02.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:02 smithi102 ceph-mon[25338]: Saving service mgr spec with placement count:2 2024-03-11T21:50:02.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:02 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:50:03.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:03 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/3588455958' entity='client.admin' 2024-03-11T21:50:04.119 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling the dashboard module... 2024-03-11T21:50:04.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:04 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1071414317' entity='client.admin' 2024-03-11T21:50:05.785 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:05 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/4147301318' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T21:50:05.785 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:05 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:50:05.785 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:05 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ignoring --setuser ceph since I am not root 2024-03-11T21:50:05.785 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:05 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ignoring --setgroup ceph since I am not root 2024-03-11T21:50:06.085 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:06 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:06.040+0000 7fa751ca3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:50:06.339 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:06 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:06.137+0000 7fa751ca3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:50:06.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:06 smithi102 ceph-mon[25338]: from='mgr.14118 172.21.15.102:0/3507526190' entity='mgr.y' 2024-03-11T21:50:06.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:06 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/4147301318' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T21:50:06.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:06 smithi102 ceph-mon[25338]: mgrmap e9: y(active, since 18s) 2024-03-11T21:50:07.123 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:06 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:06.792+0000 7fa751ca3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:50:07.123 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:06 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:06.891+0000 7fa751ca3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:50:07.123 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:07.033+0000 7fa751ca3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:50:07.123 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:07.130+0000 7fa751ca3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:50:07.311 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:50:07.311 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T21:50:07.311 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:50:07.311 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:50:07.311 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:50:07.311 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:50:07.311 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-03-11T21:50:07.311 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 9... 2024-03-11T21:50:07.417 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:07.213+0000 7fa751ca3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:50:07.702 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:07 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/4041435411' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:50:07.703 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:07.487+0000 7fa751ca3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:50:07.703 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:07.601+0000 7fa751ca3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:50:07.703 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:07.690+0000 7fa751ca3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:50:07.954 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:07.787+0000 7fa751ca3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:50:08.417 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:07.962+0000 7fa751ca3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:50:09.167 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:08 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:08.906+0000 7fa751ca3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:50:09.167 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:09 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:09.034+0000 7fa751ca3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:50:09.640 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:09 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:09.364+0000 7fa751ca3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:50:09.640 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:09 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:09.454+0000 7fa751ca3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:50:09.640 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:09 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:09.647+0000 7fa751ca3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:50:09.917 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:09 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:09.831+0000 7fa751ca3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:50:10.362 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:10 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:10.114+0000 7fa751ca3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:50:10.363 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:10 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:10.370+0000 7fa751ca3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:50:10.667 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:10 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:10.605+0000 7fa751ca3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:50:11.667 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:11 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:11.395+0000 7fa751ca3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:50:12.262 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:11 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:11.958+0000 7fa751ca3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:50:12.262 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:12.055+0000 7fa751ca3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:50:12.262 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:50:12.269+0000 7fa751ca3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:50:12.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: Active manager daemon y restarted 2024-03-11T21:50:12.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: Activating manager daemon y 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: mgrmap e10: y(active, starting, since 0.0032519s) 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: Manager daemon y is now available 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:13.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:50:13.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:50:13.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:13 smithi102 ceph-mon[25338]: mgrmap e11: y(active, since 1.00615s) 2024-03-11T21:50:14.187 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:50:14.187 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T21:50:14.187 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:50:14.187 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:50:14.187 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 9 is available 2024-03-11T21:50:14.187 INFO:teuthology.orchestra.run.smithi102.stdout:Generating a dashboard self-signed certificate... 2024-03-11T21:50:14.603 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:14 smithi102 ceph-mon[25338]: [11/Mar/2024:21:50:12] ENGINE Bus STARTING 2024-03-11T21:50:14.604 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:14 smithi102 ceph-mon[25338]: [11/Mar/2024:21:50:12] ENGINE Serving on https://172.21.15.102:7150 2024-03-11T21:50:14.604 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:14 smithi102 ceph-mon[25338]: [11/Mar/2024:21:50:13] ENGINE Serving on http://172.21.15.102:8765 2024-03-11T21:50:14.604 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:14 smithi102 ceph-mon[25338]: [11/Mar/2024:21:50:13] ENGINE Bus STARTED 2024-03-11T21:50:14.604 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:14 smithi102 ceph-mon[25338]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:50:14.604 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:14 smithi102 ceph-mon[25338]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:50:14.604 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:14 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:15.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T21:50:15.527 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial admin user... 2024-03-11T21:50:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:15 smithi102 ceph-mon[25338]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:15 smithi102 ceph-mon[25338]: mgrmap e12: y(active, since 2s) 2024-03-11T21:50:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:15 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:15 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:15 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:17.447 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:17 smithi102 ceph-mon[25338]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:17.448 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:17 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:17.672 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$PZ5Amc6YnWeWg1JLvs83LuWV39Zb671tl9/R07Oxvofk6qcU7tv1S", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710193816, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T21:50:17.672 INFO:teuthology.orchestra.run.smithi102.stdout:Fetching dashboard port number... 2024-03-11T21:50:18.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:18 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/3863682005' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T21:50:18.920 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T21:50:18.930 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:50:18.930 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-11T21:50:18.937 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:50:18.937 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-11T21:50:18.937 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-11T21:50:18.937 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T21:50:18.939 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph Dashboard is now available at: 2024-03-11T21:50:18.939 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:18.939 INFO:teuthology.orchestra.run.smithi102.stdout: URL: https://smithi102.front.sepia.ceph.com:8443/ 2024-03-11T21:50:18.939 INFO:teuthology.orchestra.run.smithi102.stdout: User: admin 2024-03-11T21:50:18.939 INFO:teuthology.orchestra.run.smithi102.stdout: Password: hf1gx0c7t2 2024-03-11T21:50:18.939 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:18.940 INFO:teuthology.orchestra.run.smithi102.stdout:Saving cluster configuration to /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config directory 2024-03-11T21:50:18.940 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling autotune for osd_memory_target 2024-03-11T21:50:19.550 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:19 smithi102 ceph-mon[25338]: mgrmap e13: y(active, since 6s) 2024-03-11T21:50:21.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T21:50:21.274 INFO:teuthology.orchestra.run.smithi102.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T21:50:21.274 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:21.274 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph telemetry on 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout:For more information see: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:21.275 INFO:teuthology.orchestra.run.smithi102.stdout:Bootstrap complete. 2024-03-11T21:50:21.317 INFO:tasks.cephadm:Fetching config... 2024-03-11T21:50:21.318 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:50:21.318 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T21:50:21.333 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T21:50:21.333 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:50:21.333 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T21:50:21.389 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T21:50:21.389 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:50:21.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T21:50:21.457 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T21:50:21.457 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:50:21.457 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T21:50:21.512 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T21:50:21.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6N/1i3OzSfShdQTUqm060yVIshIC9ep8XtunD2AMZ8N//M51PII1JqynSvbsp0Is3+lGUM6wiP60MYprP/Lb9mxz3XCP/28DN7cC1ymF2s8wkZuLGZOnRw+xF5+AFe6mN3GOMii27NN5EMfd0mtEZ1cTz5vi9+3pkWi+d60ZC+EIj1vdWl2Bun3OW2AWZHY1UbD1Oi3arGroTkbf6xYwaZ3+9K0fvKnyEGh9xbFXJaTN8wupPyV0YnT03Aj/qwKbcuPbRTLFM0hLpHLwqac91tPVbzgGpOLZZwqFHikKl93mP/Lt9dKEEcOkAnsa7TFvsz4AkIc/q2yOeS7bDxzCpeqcV44C4Psbq6iLKClrS/bfBtdX3yOB1smbbS11q9kjfJKdVl5H8bbbnd+s144gcoDkIMyDSHGZ8N+A9jHaQhQawHni3iJ/LXWtRXouBoHm+Oz/j0UkedzPCmDCpHyy3zMfNQT1K17Qu7yfR8SEYbx9pCkacSttYdwmHIiWULRE= ceph-0126e6b2-dff1-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:50:21.610 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6N/1i3OzSfShdQTUqm060yVIshIC9ep8XtunD2AMZ8N//M51PII1JqynSvbsp0Is3+lGUM6wiP60MYprP/Lb9mxz3XCP/28DN7cC1ymF2s8wkZuLGZOnRw+xF5+AFe6mN3GOMii27NN5EMfd0mtEZ1cTz5vi9+3pkWi+d60ZC+EIj1vdWl2Bun3OW2AWZHY1UbD1Oi3arGroTkbf6xYwaZ3+9K0fvKnyEGh9xbFXJaTN8wupPyV0YnT03Aj/qwKbcuPbRTLFM0hLpHLwqac91tPVbzgGpOLZZwqFHikKl93mP/Lt9dKEEcOkAnsa7TFvsz4AkIc/q2yOeS7bDxzCpeqcV44C4Psbq6iLKClrS/bfBtdX3yOB1smbbS11q9kjfJKdVl5H8bbbnd+s144gcoDkIMyDSHGZ8N+A9jHaQhQawHni3iJ/LXWtRXouBoHm+Oz/j0UkedzPCmDCpHyy3zMfNQT1K17Qu7yfR8SEYbx9pCkacSttYdwmHIiWULRE= ceph-0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:50:21.626 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6N/1i3OzSfShdQTUqm060yVIshIC9ep8XtunD2AMZ8N//M51PII1JqynSvbsp0Is3+lGUM6wiP60MYprP/Lb9mxz3XCP/28DN7cC1ymF2s8wkZuLGZOnRw+xF5+AFe6mN3GOMii27NN5EMfd0mtEZ1cTz5vi9+3pkWi+d60ZC+EIj1vdWl2Bun3OW2AWZHY1UbD1Oi3arGroTkbf6xYwaZ3+9K0fvKnyEGh9xbFXJaTN8wupPyV0YnT03Aj/qwKbcuPbRTLFM0hLpHLwqac91tPVbzgGpOLZZwqFHikKl93mP/Lt9dKEEcOkAnsa7TFvsz4AkIc/q2yOeS7bDxzCpeqcV44C4Psbq6iLKClrS/bfBtdX3yOB1smbbS11q9kjfJKdVl5H8bbbnd+s144gcoDkIMyDSHGZ8N+A9jHaQhQawHni3iJ/LXWtRXouBoHm+Oz/j0UkedzPCmDCpHyy3zMfNQT1K17Qu7yfR8SEYbx9pCkacSttYdwmHIiWULRE= ceph-0126e6b2-dff1-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:50:21.673 INFO:teuthology.orchestra.run.smithi110.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6N/1i3OzSfShdQTUqm060yVIshIC9ep8XtunD2AMZ8N//M51PII1JqynSvbsp0Is3+lGUM6wiP60MYprP/Lb9mxz3XCP/28DN7cC1ymF2s8wkZuLGZOnRw+xF5+AFe6mN3GOMii27NN5EMfd0mtEZ1cTz5vi9+3pkWi+d60ZC+EIj1vdWl2Bun3OW2AWZHY1UbD1Oi3arGroTkbf6xYwaZ3+9K0fvKnyEGh9xbFXJaTN8wupPyV0YnT03Aj/qwKbcuPbRTLFM0hLpHLwqac91tPVbzgGpOLZZwqFHikKl93mP/Lt9dKEEcOkAnsa7TFvsz4AkIc/q2yOeS7bDxzCpeqcV44C4Psbq6iLKClrS/bfBtdX3yOB1smbbS11q9kjfJKdVl5H8bbbnd+s144gcoDkIMyDSHGZ8N+A9jHaQhQawHni3iJ/LXWtRXouBoHm+Oz/j0UkedzPCmDCpHyy3zMfNQT1K17Qu7yfR8SEYbx9pCkacSttYdwmHIiWULRE= ceph-0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:50:21.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T21:50:21.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:21 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2814855109' entity='client.admin' 2024-03-11T21:50:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:24.719 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T21:50:24.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T21:50:24.953 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:25.060 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:24 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1746480101' entity='client.admin' 2024-03-11T21:50:25.060 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:25.060 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:25.060 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:50:25.060 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:25.060 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:25.060 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:27.210 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi110 2024-03-11T21:50:27.211 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:50:27.211 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.conf 2024-03-11T21:50:27.229 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:50:27.229 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:27.286 INFO:tasks.cephadm:Adding host smithi110 to orchestrator... 2024-03-11T21:50:27.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch host add smithi110 2024-03-11T21:50:27.527 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:27.810 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:27 smithi102 ceph-mon[25338]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:27.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:27.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:27.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:27.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:27.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:27.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:27.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:28.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:28 smithi102 ceph-mon[25338]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-11T21:50:28.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:28 smithi102 ceph-mon[25338]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:28.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:28 smithi102 ceph-mon[25338]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:28.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:28 smithi102 ceph-mon[25338]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:29.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:29 smithi102 ceph-mon[25338]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:30.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:30 smithi102 ceph-mon[25338]: Deploying cephadm binary to smithi110 2024-03-11T21:50:33.332 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi110' with addr '172.21.15.110' 2024-03-11T21:50:33.619 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:33 smithi102 ceph-mon[25338]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:33.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T21:50:34.115 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:34.523 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:34 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:34.523 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:34 smithi102 ceph-mon[25338]: Added host smithi110 2024-03-11T21:50:34.523 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:34 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:34.523 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:34 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:35.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:35 smithi102 ceph-mon[25338]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:35.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:35 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:35.754 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:50:35.755 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}] 2024-03-11T21:50:36.283 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T21:50:36.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T21:50:36.514 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:37.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:36 smithi102 ceph-mon[25338]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:37.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:36 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:37.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:36 smithi102 ceph-mon[25338]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:38.102 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-03-11T21:50:38.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:38 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1166618157' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T21:50:38.720 INFO:tasks.cephadm:Adding mon.a on smithi102 2024-03-11T21:50:38.721 INFO:tasks.cephadm:Adding mon.c on smithi102 2024-03-11T21:50:38.721 INFO:tasks.cephadm:Adding mon.b on smithi110 2024-03-11T21:50:38.721 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi110:172.21.15.110=b' 2024-03-11T21:50:39.381 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T21:50:39.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:39 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1166618157' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T21:50:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:39 smithi102 ceph-mon[25338]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:50:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:39 smithi102 ceph-mon[25338]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T21:50:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:50:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: Updating smithi110:/etc/ceph/ceph.conf 2024-03-11T21:50:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:41.255 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled mon update... 2024-03-11T21:50:41.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:50:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:41.811 DEBUG:teuthology.orchestra.run.smithi102:mon.c> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T21:50:41.813 DEBUG:teuthology.orchestra.run.smithi110:mon.b> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T21:50:41.815 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:50:41.816 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:50:42.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:42 smithi102 ceph-mon[25338]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi110:172.21.15.110=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:42.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:42 smithi102 ceph-mon[25338]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi110:172.21.15.110=b;count:3 2024-03-11T21:50:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:42 smithi102 ceph-mon[25338]: Deploying daemon mon.b on smithi110 2024-03-11T21:50:42.743 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:43.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:43 smithi102 ceph-mon[25338]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:45.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:45 smithi102 ceph-mon[25338]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:45.808 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:50:45.808 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":1,"fsid":"0126e6b2-dff1-11ee-95c8-87774f69a715","modified":"2024-03-11T21:49:11.989724Z","created":"2024-03-11T21:49:11.989724Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T21:50:45.809 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 1 2024-03-11T21:50:47.499 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:50:47.499 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:50:47.742 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:50:51.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:50 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:50:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: mon.a calling monitor election 2024-03-11T21:50:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: mon.b calling monitor election 2024-03-11T21:50:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:50:51.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: monmap e2: 2 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:50:51.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: fsmap 2024-03-11T21:50:51.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:50:51.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: mgrmap e13: y(active, since 38s) 2024-03-11T21:50:51.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: overall HEALTH_OK 2024-03-11T21:50:51.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:51.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:51.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:51.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:50:51.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:52.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:51 smithi102 ceph-mon[25338]: Deploying daemon mon.c on smithi102 2024-03-11T21:50:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:51 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:52.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:51 smithi110 ceph-mon[30535]: Deploying daemon mon.c on smithi102 2024-03-11T21:50:52.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:51 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:52.491 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:50:52.492 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":2,"fsid":"0126e6b2-dff1-11ee-95c8-87774f69a715","modified":"2024-03-11T21:50:45.859956Z","created":"2024-03-11T21:49:11.989724Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T21:50:52.494 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 2 2024-03-11T21:50:53.095 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:52 smithi110 ceph-mon[30535]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:53.095 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:52 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/3102282491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:50:53.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:52 smithi102 ceph-mon[25338]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:53.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:52 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/3102282491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:50:53.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:53 smithi102 systemd[1]: Starting Ceph mon.c for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:50:54.121 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:50:54.122 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:50:54.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 podman[34158]: 2024-03-11 21:50:53.921602913 +0000 UTC m=+0.017758071 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:50:54.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 podman[34158]: 2024-03-11 21:50:54.026190995 +0000 UTC m=+0.122346189 container create d65f8f2d5f2ef62ff315196447cc826cb3d6b6bb1927c9e0674e96fd9f6af839 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T21:50:54.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 podman[34158]: 2024-03-11 21:50:54.253398332 +0000 UTC m=+0.349553488 container init d65f8f2d5f2ef62ff315196447cc826cb3d6b6bb1927c9e0674e96fd9f6af839 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-11T21:50:54.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 podman[34158]: 2024-03-11 21:50:54.256333002 +0000 UTC m=+0.352488151 container start d65f8f2d5f2ef62ff315196447cc826cb3d6b6bb1927c9e0674e96fd9f6af839 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-03-11T21:50:54.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:50:54.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T21:50:54.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: pidfile_write: ignore empty --pid-file 2024-03-11T21:50:54.371 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:50:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: load: jerasure load: lrc 2024-03-11T21:50:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:50:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Git sha 0 2024-03-11T21:50:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T21:50:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: DB SUMMARY 2024-03-11T21:50:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: DB Session ID: 5SQNSU6OP2CMT40DACNT 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.env: 0x55886e1a3ae0 2024-03-11T21:50:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.info_log: 0x5588708238c0 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.statistics: (nil) 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.use_fsync: 0 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:50:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.db_log_dir: 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.wal_dir: 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.write_buffer_manager: 0x5588707f7400 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:50:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.unordered_write: 0 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.row_cache: None 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.wal_filter: None 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:50:54.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.wal_compression: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:50:54.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:50:54.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:50:54.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:50:54.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:50:54.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:50:54.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:50:54.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:50:54.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_open_files: -1 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Compression algorithms supported: 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: kZSTD supported: 0 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:50:54.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.merge_operator: 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_filter: None 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:50:54.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5588708239c0) 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_type: 0 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_index_type: 0 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_shortening: 1 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: checksum: 4 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: no_block_cache: 0 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache: 0x558870827090 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_options: 2024-03-11T21:50:54.678 INFO:journalctl@ceph.mon.c.smithi102.stdout: capacity : 536870912 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_shard_bits : 4 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: strict_capacity_limit : 0 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_compressed: (nil) 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: persistent_cache: (nil) 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size: 4096 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size_deviation: 10 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_restart_interval: 16 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_block_restart_interval: 1 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: metadata_block_size: 4096 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: partition_filters: 0 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: use_delta_encoding: 1 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: filter_policy: bloomfilter 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: whole_key_filtering: 1 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: verify_compression: 0 2024-03-11T21:50:54.679 INFO:journalctl@ceph.mon.c.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout: format_version: 5 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_index_compression: 1 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_align: 0 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout: max_auto_readahead_size: 262144 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout: prepopulate_block_cache: 0 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression: NoCompression 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.num_levels: 7 2024-03-11T21:50:54.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:50:54.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:50:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:50:54.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:50:54.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.ttl: 2592000 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:50:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a1f46ddf-22f8-4ad4-92be-d9057f965278 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193854286141, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193854286543, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710193854, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a1f46ddf-22f8-4ad4-92be-d9057f965278", "db_session_id": "5SQNSU6OP2CMT40DACNT", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193854286622, "job": 1, "event": "recovery_finished"} 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:50:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558870918000 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: DB pointer 0x558870902000 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** DB Stats ** 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:50:54.687 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-11T21:50:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative compaction: 0.00 GB write, 0.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval compaction: 0.00 GB write, 0.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache BinnedLRUCache@0x558870827090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 7e-06 secs_since: 0 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: using public_addrv [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] 2024-03-11T21:50:54.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: starting mon.c rank -1 at public addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] at bind addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(???) e0 preinit fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout: e1 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout: legacy client fscid: -1 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout: No filesystems configured 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T21:50:54.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: Updating smithi110:/etc/ceph/ceph.conf 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi110:172.21.15.110=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi110:172.21.15.110=b;count:3 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: Deploying daemon mon.b on smithi110 2024-03-11T21:50:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.a calling monitor election 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.b calling monitor election 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:50:54.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: monmap e2: 2 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: fsmap 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mgrmap e13: y(active, since 38s) 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: overall HEALTH_OK 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: Deploying daemon mon.c on smithi102 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/3102282491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:50:54.694 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 ceph-mon[34177]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T21:50:54.695 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 bash[34158]: d65f8f2d5f2ef62ff315196447cc826cb3d6b6bb1927c9e0674e96fd9f6af839 2024-03-11T21:50:54.695 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:50:54 smithi102 systemd[1]: Started Ceph mon.c for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:50:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:50:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: mon.a calling monitor election 2024-03-11T21:50:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: mon.b calling monitor election 2024-03-11T21:50:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: fsmap 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:50:59.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: mgrmap e13: y(active, since 47s) 2024-03-11T21:50:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-03-11T21:50:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-11T21:50:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-11T21:50:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: mon.c (rank 2) addr [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] is down (out of quorum) 2024-03-11T21:50:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:50:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:59.724 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-11T21:50:59.724 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":3,"fsid":"0126e6b2-dff1-11ee-95c8-87774f69a715","modified":"2024-03-11T21:50:54.316735Z","created":"2024-03-11T21:49:11.989724Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3301","nonce":0},{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: mon.a calling monitor election 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: mon.b calling monitor election 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: fsmap 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: mgrmap e13: y(active, since 47s) 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: mon.c (rank 2) addr [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] is down (out of quorum) 2024-03-11T21:50:59.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:59.727 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:59.727 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:59.727 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:50:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:50:59.727 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 3 2024-03-11T21:51:00.311 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T21:51:00.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T21:51:00.603 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:00 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:00.603 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:00 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/2326516875' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:51:00.603 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:00 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:00.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:00 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:00 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/2326516875' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:51:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:00 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:00.740 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: mon.a calling monitor election 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: mon.b calling monitor election 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: fsmap 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: mgrmap e13: y(active, since 47s) 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: mon.c (rank 2) addr [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] is down (out of quorum) 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:01.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:01.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:01.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/2326516875' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:51:01.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:02.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: mon.c calling monitor election 2024-03-11T21:51:02.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-11T21:51:02.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: Updating smithi110:/etc/ceph/ceph.conf 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: mon.c calling monitor election 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: mon.b calling monitor election 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: mon.a calling monitor election 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: fsmap 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: mgrmap e13: y(active, since 49s) 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: Cluster is now healthy 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: overall HEALTH_OK 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:51:02.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:51:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: mon.c calling monitor election 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: Updating smithi110:/etc/ceph/ceph.conf 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: mon.c calling monitor election 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: mon.b calling monitor election 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: mon.a calling monitor election 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: fsmap 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: mgrmap e13: y(active, since 49s) 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-11T21:51:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: Cluster is now healthy 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: overall HEALTH_OK 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:51:02.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:02.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: mon.c calling monitor election 2024-03-11T21:51:02.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: Updating smithi110:/etc/ceph/ceph.conf 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: mon.c calling monitor election 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: mon.b calling monitor election 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: mon.a calling monitor election 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: fsmap 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: mgrmap e13: y(active, since 49s) 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: Cluster is now healthy 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: overall HEALTH_OK 2024-03-11T21:51:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:51:02.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:02.592 INFO:teuthology.orchestra.run.smithi102.stdout:# minimal ceph.conf for 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:51:02.593 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-03-11T21:51:02.593 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:51:02.593 INFO:teuthology.orchestra.run.smithi102.stdout: mon_host = [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] 2024-03-11T21:51:03.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[25338]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:51:03.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[25338]: Reconfiguring daemon mon.a on smithi102 2024-03-11T21:51:03.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[25338]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:03.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:03.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:03.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2091108645' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:03.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[34177]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:51:03.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[34177]: Reconfiguring daemon mon.a on smithi102 2024-03-11T21:51:03.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[34177]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:03.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:03.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:03.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:03 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/2091108645' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:03.192 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T21:51:03.192 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:51:03.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:51:03.222 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:51:03.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:51:03.290 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:51:03.290 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:51:03.324 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:51:03.325 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:51:03.389 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:03 smithi110 ceph-mon[30535]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:51:03.389 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:03 smithi110 ceph-mon[30535]: Reconfiguring daemon mon.a on smithi102 2024-03-11T21:51:03.389 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:03 smithi110 ceph-mon[30535]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:03.389 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:03 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:51:03.389 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:03 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:03.389 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:03 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/2091108645' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:03.410 INFO:tasks.cephadm:Adding mgr.y on smithi102 2024-03-11T21:51:03.410 INFO:tasks.cephadm:Adding mgr.x on smithi110 2024-03-11T21:51:03.411 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi102=y;smithi110=x' 2024-03-11T21:51:03.685 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:51:05.131 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:05 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:05.131 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:05 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:05.131 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:05 smithi110 ceph-mon[30535]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:51:05.131 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:05 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:51:05.132 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:05 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:51:05.132 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:05 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:05.132 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:05 smithi110 ceph-mon[30535]: Reconfiguring daemon mon.c on smithi102 2024-03-11T21:51:05.132 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:05 smithi110 ceph-mon[30535]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[25338]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[25338]: Reconfiguring daemon mon.c on smithi102 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[25338]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[34177]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:51:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:51:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:51:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[34177]: Reconfiguring daemon mon.c on smithi102 2024-03-11T21:51:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:05 smithi102 ceph-mon[34177]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:05.450 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled mgr update... 2024-03-11T21:51:05.970 DEBUG:teuthology.orchestra.run.smithi110:mgr.x> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T21:51:05.972 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T21:51:05.973 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:51:05.973 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:51:05.991 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:51:05.991 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme1n1 2024-03-11T21:51:06.049 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme1n1 2024-03-11T21:51:06.050 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:51:06.050 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T21:51:06.050 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:51:06.050 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:51:06.050 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-11 21:50:22.248328353 +0000 2024-03-11T21:51:06.050 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-11 21:47:46.986631716 +0000 2024-03-11T21:51:06.050 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-11 21:47:46.986631716 +0000 2024-03-11T21:51:06.050 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-11 21:47:46.974631961 +0000 2024-03-11T21:51:06.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:51:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-11T21:51:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-11T21:51:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000332994 s, 1.5 MB/s 2024-03-11T21:51:06.123 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:51:06.181 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme2n1 2024-03-11T21:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme2n1 2024-03-11T21:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-11T21:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-11 21:50:22.258328149 +0000 2024-03-11T21:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-11 21:47:47.130628776 +0000 2024-03-11T21:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-11 21:47:47.130628776 +0000 2024-03-11T21:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-11 21:47:47.119629000 +0000 2024-03-11T21:51:06.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:51:06.309 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-11T21:51:06.309 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-11T21:51:06.310 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000225146 s, 2.3 MB/s 2024-03-11T21:51:06.311 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:51:06.370 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme3n1 2024-03-11T21:51:06.427 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme3n1 2024-03-11T21:51:06.428 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:51:06.428 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 788 Links: 1 Device type: 103,6 2024-03-11T21:51:06.428 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:51:06.428 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:51:06.428 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-11 21:50:22.268327946 +0000 2024-03-11T21:51:06.428 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-11 21:47:47.405623160 +0000 2024-03-11T21:51:06.428 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-11 21:47:47.405623160 +0000 2024-03-11T21:51:06.428 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-11 21:47:47.398623303 +0000 2024-03-11T21:51:06.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:51:06.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:06 smithi110 ceph-mon[30535]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi110=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:51:06.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:06 smithi110 ceph-mon[30535]: Saving service mgr spec with placement smithi102=y;smithi110=x;count:2 2024-03-11T21:51:06.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:06 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:06.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:06 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:06.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:06 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:06.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:06 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:51:06.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:06 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:51:06.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:06 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:06.497 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-11T21:51:06.497 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-11T21:51:06.497 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000244106 s, 2.1 MB/s 2024-03-11T21:51:06.499 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:51:06.558 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme4n1 2024-03-11T21:51:06.616 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme4n1 2024-03-11T21:51:06.616 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:51:06.616 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 800 Links: 1 Device type: 103,8 2024-03-11T21:51:06.617 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:51:06.617 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:51:06.617 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-11 21:50:22.278327742 +0000 2024-03-11T21:51:06.617 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-11 21:47:47.556620077 +0000 2024-03-11T21:51:06.617 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-11 21:47:47.556620077 +0000 2024-03-11T21:51:06.617 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-11 21:47:47.548620240 +0000 2024-03-11T21:51:06.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:51:06.686 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-11T21:51:06.686 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-11T21:51:06.686 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000252066 s, 2.0 MB/s 2024-03-11T21:51:06.688 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:51:06.746 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:51:06.746 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:51:06.762 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:51:06.763 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/nvme1n1 2024-03-11T21:51:06.817 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/nvme1n1 2024-03-11T21:51:06.817 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:51:06.817 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T21:51:06.817 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:51:06.817 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:51:06.817 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-11 21:50:38.123813841 +0000 2024-03-11T21:51:06.817 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-11 21:47:48.740092672 +0000 2024-03-11T21:51:06.817 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-11 21:47:48.740092672 +0000 2024-03-11T21:51:06.817 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-11 21:47:48.730092853 +0000 2024-03-11T21:51:06.817 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:51:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-11T21:51:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-11T21:51:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000127923 s, 4.0 MB/s 2024-03-11T21:51:06.885 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:51:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[25338]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi110=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:51:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[25338]: Saving service mgr spec with placement smithi102=y;smithi110=x;count:2 2024-03-11T21:51:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:51:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:51:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:06.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[34177]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi110=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:51:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[34177]: Saving service mgr spec with placement smithi102=y;smithi110=x;count:2 2024-03-11T21:51:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:51:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:51:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:06 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:06.940 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/nvme2n1 2024-03-11T21:51:06.995 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/nvme2n1 2024-03-11T21:51:06.996 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:51:06.996 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T21:51:06.996 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:51:06.996 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:51:06.996 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-11 21:50:38.131813696 +0000 2024-03-11T21:51:06.996 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-11 21:47:48.876090214 +0000 2024-03-11T21:51:06.996 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-11 21:47:48.876090214 +0000 2024-03-11T21:51:06.996 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-11 21:47:48.873090268 +0000 2024-03-11T21:51:06.996 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:51:07.061 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-11T21:51:07.062 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-11T21:51:07.062 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000107375 s, 4.8 MB/s 2024-03-11T21:51:07.063 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:51:07.118 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/nvme3n1 2024-03-11T21:51:07.173 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/nvme3n1 2024-03-11T21:51:07.173 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:51:07.173 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T21:51:07.173 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:51:07.173 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:51:07.173 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-11 21:50:38.140813534 +0000 2024-03-11T21:51:07.173 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-11 21:47:49.026087503 +0000 2024-03-11T21:51:07.173 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-11 21:47:49.026087503 +0000 2024-03-11T21:51:07.173 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-11 21:47:49.017087665 +0000 2024-03-11T21:51:07.174 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:51:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-11T21:51:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-11T21:51:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000108365 s, 4.7 MB/s 2024-03-11T21:51:07.240 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:51:07.297 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/nvme4n1 2024-03-11T21:51:07.353 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/nvme4n1 2024-03-11T21:51:07.353 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:51:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T21:51:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:51:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:51:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-11 21:50:38.148813389 +0000 2024-03-11T21:51:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-11 21:47:49.177084773 +0000 2024-03-11T21:51:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-11 21:47:49.177084773 +0000 2024-03-11T21:51:07.354 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-11 21:47:49.171084882 +0000 2024-03-11T21:51:07.354 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:51:07.420 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-11T21:51:07.420 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-11T21:51:07.420 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000136974 s, 3.7 MB/s 2024-03-11T21:51:07.421 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:51:07.477 INFO:tasks.cephadm:Deploying osd.0 on smithi102 with /dev/nvme4n1... 2024-03-11T21:51:07.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:51:07.727 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:07.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:51:07.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: Reconfiguring daemon mon.b on smithi110 2024-03-11T21:51:07.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:07.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:07.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:07.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:51:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:51:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:51:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:07.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:51:07.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: Reconfiguring daemon mon.b on smithi110 2024-03-11T21:51:07.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:07.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:07.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:07.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:07.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:07.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:07.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: Reconfiguring daemon mon.b on smithi110 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:07.830 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:07.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:07.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:07.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:07.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:51:07.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:51:07.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:51:07.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:08.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:08 smithi110 ceph-mon[30535]: Deploying daemon mgr.x on smithi110 2024-03-11T21:51:08.803 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:08 smithi102 ceph-mon[25338]: Deploying daemon mgr.x on smithi110 2024-03-11T21:51:08.803 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:08 smithi102 ceph-mon[34177]: Deploying daemon mgr.x on smithi110 2024-03-11T21:51:09.437 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:51:09.437 INFO:teuthology.orchestra.run.smithi102.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:51:09.437 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:51:09.437 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-11T21:51:09.437 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-11T21:51:09.437 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012299 s, 853 MB/s 2024-03-11T21:51:09.437 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-11T21:51:09.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:09 smithi110 ceph-mon[30535]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:09.737 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:09 smithi110 systemd[1]: Started Ceph mgr.x for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:51:09.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:09 smithi102 ceph-mon[25338]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:09.832 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:09 smithi102 ceph-mon[34177]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:10.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi102:/dev/nvme4n1 2024-03-11T21:51:10.303 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:10.487 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:10 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:10.122+0000 7f6665cc4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:51:10.487 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:10 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:10.218+0000 7f6665cc4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:51:10.907 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:10 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:10.908 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:10 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:10.908 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:10 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:10.908 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:10 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:10.908 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:10 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:10.908 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:10 smithi110 ceph-mon[30535]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:11.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:11.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:11.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:11.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:11.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:11.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[25338]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:11.065 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:11.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:11.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:11.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:11.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:11.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:10 smithi102 ceph-mon[34177]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:11.169 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:10 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:10.914+0000 7f6665cc4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:51:11.169 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:11 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:11.006+0000 7f6665cc4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:51:11.487 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:11 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:11.176+0000 7f6665cc4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:51:11.487 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:11 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:11.274+0000 7f6665cc4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:51:11.487 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:11 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:11.356+0000 7f6665cc4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:51:11.897 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:11 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:11.616+0000 7f6665cc4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:51:11.897 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:11 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:11.725+0000 7f6665cc4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:51:11.897 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:11 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:11.810+0000 7f6665cc4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:51:11.897 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:11 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:11.904+0000 7f6665cc4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:51:12.237 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:12.079+0000 7f6665cc4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:51:12.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:12.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: Reconfiguring daemon mgr.y on smithi102 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:51:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:51:12.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:12.322 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:12.322 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:12.322 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: Reconfiguring daemon mgr.y on smithi102 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:12.323 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: Reconfiguring daemon mgr.y on smithi102 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:51:12.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:13.343 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:13.040+0000 7f6665cc4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:51:13.343 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:13.172+0000 7f6665cc4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[25338]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[25338]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[34177]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:51:13.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[34177]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:13.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:13.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:13.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:13.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.737 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:13.502+0000 7f6665cc4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:51:13.737 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:13.591+0000 7f6665cc4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:51:13.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-mon[30535]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:51:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-mon[30535]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:14.048 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:13.784+0000 7f6665cc4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:51:14.048 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:13 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:13.967+0000 7f6665cc4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:51:14.487 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:14.253+0000 7f6665cc4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:51:14.987 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:14.521+0000 7f6665cc4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:51:14.987 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:14.763+0000 7f6665cc4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:51:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[25338]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/677002580' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f36f4f8-fe18-4d44-b3df-b07709f4af70"}]: dispatch 2024-03-11T21:51:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/677002580' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f36f4f8-fe18-4d44-b3df-b07709f4af70"}]': finished 2024-03-11T21:51:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[25338]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:51:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:15.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[34177]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:15.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/677002580' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f36f4f8-fe18-4d44-b3df-b07709f4af70"}]: dispatch 2024-03-11T21:51:15.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/677002580' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f36f4f8-fe18-4d44-b3df-b07709f4af70"}]': finished 2024-03-11T21:51:15.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[34177]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:51:15.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:15 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:15.737 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:15 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:15.556+0000 7f6665cc4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:51:15.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:15 smithi110 ceph-mon[30535]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:15.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:15 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/677002580' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f36f4f8-fe18-4d44-b3df-b07709f4af70"}]: dispatch 2024-03-11T21:51:15.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:15 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/677002580' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f36f4f8-fe18-4d44-b3df-b07709f4af70"}]': finished 2024-03-11T21:51:15.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:15 smithi110 ceph-mon[30535]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:51:15.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:15 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:16.427 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:16 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1953285345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:51:16.427 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:16 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:16.122+0000 7f6665cc4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:51:16.427 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:16 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:16.219+0000 7f6665cc4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:51:16.427 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:51:16 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:51:16.434+0000 7f6665cc4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:51:16.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:16 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1953285345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:51:16.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:16 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1953285345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:51:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[25338]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[25338]: Standby manager daemon x started 2024-03-11T21:51:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[25338]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:51:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[25338]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:51:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[25338]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:51:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[25338]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:51:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[34177]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[34177]: Standby manager daemon x started 2024-03-11T21:51:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[34177]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:51:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[34177]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:51:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[34177]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:51:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:17 smithi102 ceph-mon[34177]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:51:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:17 smithi110 ceph-mon[30535]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:17 smithi110 ceph-mon[30535]: Standby manager daemon x started 2024-03-11T21:51:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:17 smithi110 ceph-mon[30535]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:51:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:17 smithi110 ceph-mon[30535]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:51:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:17 smithi110 ceph-mon[30535]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:51:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:17 smithi110 ceph-mon[30535]: from='mgr.? 172.21.15.110:0/221052810' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:51:18.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:18 smithi102 ceph-mon[34177]: mgrmap e14: y(active, since 65s), standbys: x 2024-03-11T21:51:18.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:18 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:51:18.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:18 smithi102 ceph-mon[25338]: mgrmap e14: y(active, since 65s), standbys: x 2024-03-11T21:51:18.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:18 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:51:18.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:18 smithi110 ceph-mon[30535]: mgrmap e14: y(active, since 65s), standbys: x 2024-03-11T21:51:18.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:18 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:51:19.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:19 smithi102 ceph-mon[25338]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:19.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:19 smithi102 ceph-mon[34177]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:19.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:19 smithi110 ceph-mon[30535]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:21 smithi102 ceph-mon[25338]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:21 smithi102 ceph-mon[34177]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:21.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:21 smithi110 ceph-mon[30535]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:22.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:22 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:51:22.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:22 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:22.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:22 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:51:22.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:22 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:22.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:22 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:51:22.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:22 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:23.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:23 smithi102 ceph-mon[25338]: Deploying daemon osd.0 on smithi102 2024-03-11T21:51:23.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:23 smithi102 ceph-mon[25338]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:23.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:23 smithi102 ceph-mon[34177]: Deploying daemon osd.0 on smithi102 2024-03-11T21:51:23.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:23 smithi102 ceph-mon[34177]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:23.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:23 smithi110 ceph-mon[30535]: Deploying daemon osd.0 on smithi102 2024-03-11T21:51:23.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:23 smithi110 ceph-mon[30535]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:25.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:25 smithi102 ceph-mon[25338]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:25.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:25 smithi102 ceph-mon[34177]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:25.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:25 smithi110 ceph-mon[30535]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:27.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:27 smithi102 ceph-mon[25338]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:27.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:27.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:27.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:27.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:27 smithi102 ceph-mon[34177]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:27.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:27 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:27.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:27 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:27.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:27 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:27.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:27 smithi110 ceph-mon[30535]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:27.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:27 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:27.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:27 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:27.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:27 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:29.527 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:29 smithi102 ceph-mon[25338]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:29.528 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:29 smithi102 ceph-mon[34177]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:29.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:29 smithi110 ceph-mon[30535]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:30.791 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 0 on host 'smithi102' 2024-03-11T21:51:30.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:30.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:30.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:30 smithi110 ceph-mon[30535]: from='osd.0 [v2:172.21.15.102:6802/1026485852,v1:172.21.15.102:6803/1026485852]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:51:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:30 smithi110 ceph-mon[30535]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:51:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:30 smithi110 ceph-mon[30535]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:31.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:31.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:31.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[25338]: from='osd.0 [v2:172.21.15.102:6802/1026485852,v1:172.21.15.102:6803/1026485852]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:51:31.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[25338]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:51:31.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[25338]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:31.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:31.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:31.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:31.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:31.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:31.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[34177]: from='osd.0 [v2:172.21.15.102:6802/1026485852,v1:172.21.15.102:6803/1026485852]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:51:31.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[34177]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:51:31.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[34177]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:31.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:31.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:31.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:31.437 DEBUG:teuthology.orchestra.run.smithi102:osd.0> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T21:51:31.439 INFO:tasks.cephadm:Deploying osd.1 on smithi102 with /dev/nvme3n1... 2024-03-11T21:51:31.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-11T21:51:31.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:31 smithi110 ceph-mon[30535]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:51:31.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:31 smithi110 ceph-mon[30535]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:51:31.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:31 smithi110 ceph-mon[30535]: from='osd.0 [v2:172.21.15.102:6802/1026485852,v1:172.21.15.102:6803/1026485852]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:51:31.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:31 smithi110 ceph-mon[30535]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:51:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:32.126 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:32.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[25338]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:51:32.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[25338]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:51:32.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[25338]: from='osd.0 [v2:172.21.15.102:6802/1026485852,v1:172.21.15.102:6803/1026485852]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:51:32.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:32.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[25338]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:51:32.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:32.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:32.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[34177]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:51:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[34177]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:51:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[34177]: from='osd.0 [v2:172.21.15.102:6802/1026485852,v1:172.21.15.102:6803/1026485852]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:51:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[34177]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:51:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:32.169 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0[38048]: 2024-03-11T21:51:31.723+0000 7f05f89de700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T21:51:32.169 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:51:31 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0[38048]: 2024-03-11T21:51:31.725+0000 7f05f47d1700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:51:32.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[34177]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:51:32.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[34177]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:51:32.871 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:32.871 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:32.871 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[34177]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:32.871 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[25338]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:51:32.871 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[25338]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:51:32.871 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:32.871 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:32.871 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:32 smithi102 ceph-mon[25338]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:32.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:32 smithi110 ceph-mon[30535]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:51:32.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:32 smithi110 ceph-mon[30535]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:51:32.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:32 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:32.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:32 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:32.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:32 smithi110 ceph-mon[30535]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:51:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[25338]: purged_snaps scrub starts 2024-03-11T21:51:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[25338]: purged_snaps scrub ok 2024-03-11T21:51:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[25338]: osd.0 [v2:172.21.15.102:6802/1026485852,v1:172.21.15.102:6803/1026485852] boot 2024-03-11T21:51:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[25338]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:51:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[34177]: purged_snaps scrub starts 2024-03-11T21:51:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[34177]: purged_snaps scrub ok 2024-03-11T21:51:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[34177]: osd.0 [v2:172.21.15.102:6802/1026485852,v1:172.21.15.102:6803/1026485852] boot 2024-03-11T21:51:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[34177]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:51:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:33 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:33.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:33 smithi110 ceph-mon[30535]: purged_snaps scrub starts 2024-03-11T21:51:33.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:33 smithi110 ceph-mon[30535]: purged_snaps scrub ok 2024-03-11T21:51:33.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:33 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:33.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:33 smithi110 ceph-mon[30535]: osd.0 [v2:172.21.15.102:6802/1026485852,v1:172.21.15.102:6803/1026485852] boot 2024-03-11T21:51:33.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:33 smithi110 ceph-mon[30535]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:51:33.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:33 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:51:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/nvme3n1 2024-03-11T21:51:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:51:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-11T21:51:34.896 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-11T21:51:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-11T21:51:34.896 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129716 s, 808 MB/s 2024-03-11T21:51:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-11T21:51:34.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:34 smithi110 ceph-mon[30535]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:51:34.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:34 smithi110 ceph-mon[30535]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:35.105 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:34 smithi102 ceph-mon[25338]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:51:35.106 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:34 smithi102 ceph-mon[25338]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:35.106 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:34 smithi102 ceph-mon[34177]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:51:35.106 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:34 smithi102 ceph-mon[34177]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:35.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi102:/dev/nvme3n1 2024-03-11T21:51:36.280 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:37.314 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: Detected new or changed devices on smithi102 2024-03-11T21:51:37.314 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.314 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.314 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:51:37.314 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi102 to 16064M 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[34177]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: Detected new or changed devices on smithi102 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi102 to 16064M 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:37.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.316 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:37 smithi102 ceph-mon[25338]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:37.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: Detected new or changed devices on smithi102 2024-03-11T21:51:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:51:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi102 to 16064M 2024-03-11T21:51:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:37 smithi110 ceph-mon[30535]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:38.581 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:38 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:51:38.581 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:38 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:51:38.581 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:38 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:38.582 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:38 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:51:38.582 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:38 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:51:38.582 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:38 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:38.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:38 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:51:38.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:38 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:51:38.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:38 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:39 smithi102 ceph-mon[25338]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:51:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:39 smithi102 ceph-mon[25338]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:39.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:39 smithi102 ceph-mon[34177]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:51:39.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:39 smithi102 ceph-mon[34177]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:39.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:39 smithi110 ceph-mon[30535]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:51:39.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:39 smithi110 ceph-mon[30535]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[25338]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/3263994405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd35249-7d61-4d04-ba14-31debbf69fd5"}]: dispatch 2024-03-11T21:51:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd35249-7d61-4d04-ba14-31debbf69fd5"}]: dispatch 2024-03-11T21:51:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9bd35249-7d61-4d04-ba14-31debbf69fd5"}]': finished 2024-03-11T21:51:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[25338]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:51:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[34177]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/3263994405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd35249-7d61-4d04-ba14-31debbf69fd5"}]: dispatch 2024-03-11T21:51:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd35249-7d61-4d04-ba14-31debbf69fd5"}]: dispatch 2024-03-11T21:51:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9bd35249-7d61-4d04-ba14-31debbf69fd5"}]': finished 2024-03-11T21:51:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[34177]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:51:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:41 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:41.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:41 smithi110 ceph-mon[30535]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:41.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:41 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/3263994405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd35249-7d61-4d04-ba14-31debbf69fd5"}]: dispatch 2024-03-11T21:51:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:41 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd35249-7d61-4d04-ba14-31debbf69fd5"}]: dispatch 2024-03-11T21:51:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:41 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9bd35249-7d61-4d04-ba14-31debbf69fd5"}]': finished 2024-03-11T21:51:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:41 smithi110 ceph-mon[30535]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:51:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:41 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:42 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/991893271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:51:42.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:42 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/991893271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:51:42.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:42 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/991893271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:51:43.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:43 smithi102 ceph-mon[25338]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:43.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:43 smithi102 ceph-mon[34177]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:43.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:43 smithi110 ceph-mon[30535]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:45.561 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:45 smithi102 ceph-mon[25338]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:45.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:45 smithi102 ceph-mon[34177]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:45.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:45 smithi110 ceph-mon[30535]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:47.563 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:47 smithi102 ceph-mon[34177]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:47.563 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:47 smithi102 ceph-mon[25338]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:47.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:47 smithi110 ceph-mon[30535]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:48.525 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:51:48.525 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:48.525 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:51:48.525 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:48.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:51:48.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:49.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:49 smithi102 ceph-mon[25338]: Deploying daemon osd.1 on smithi102 2024-03-11T21:51:49.623 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:49 smithi102 ceph-mon[25338]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:49.623 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:49 smithi102 ceph-mon[34177]: Deploying daemon osd.1 on smithi102 2024-03-11T21:51:49.623 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:49 smithi102 ceph-mon[34177]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:49.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:49 smithi110 ceph-mon[30535]: Deploying daemon osd.1 on smithi102 2024-03-11T21:51:49.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:49 smithi110 ceph-mon[30535]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:51 smithi102 ceph-mon[25338]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:51.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:51 smithi102 ceph-mon[34177]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:51.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:51 smithi110 ceph-mon[30535]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:53.534 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:53 smithi102 ceph-mon[25338]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:53.534 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:53.534 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:53.534 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:53.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:53 smithi102 ceph-mon[34177]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:53.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:53.535 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:53.535 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:53.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:53 smithi110 ceph-mon[30535]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:53.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:53.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:53.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:55 smithi102 ceph-mon[25338]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:55.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:55 smithi102 ceph-mon[34177]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:55.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:55 smithi110 ceph-mon[30535]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:56.784 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 1 on host 'smithi102' 2024-03-11T21:51:57.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:57.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:57.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:51:57.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:57.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:56 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:51:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:56 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:57.391 DEBUG:teuthology.orchestra.run.smithi102:osd.1> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T21:51:57.393 INFO:tasks.cephadm:Deploying osd.2 on smithi102 with /dev/nvme2n1... 2024-03-11T21:51:57.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-11T21:51:57.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-mon[25338]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:51:57.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-mon[25338]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:51:57.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-mon[25338]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:51:57.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:57.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-mon[34177]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:51:57.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-mon[34177]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:51:57.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-mon[34177]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:51:57.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:57.919 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1[45431]: 2024-03-11T21:51:57.817+0000 7facf576f700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T21:51:57.919 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:51:57 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1[45431]: 2024-03-11T21:51:57.820+0000 7facf1562700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:51:58.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:57 smithi110 ceph-mon[30535]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:51:58.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:57 smithi110 ceph-mon[30535]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:51:58.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:57 smithi110 ceph-mon[30535]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:51:58.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:57 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:58.564 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:59.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[25338]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:51:59.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[25338]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:51:59.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:59.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:59.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[25338]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:59.098 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[34177]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:51:59.098 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[34177]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:51:59.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:59.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:59.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:58 smithi102 ceph-mon[34177]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:51:59.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:58 smithi110 ceph-mon[30535]: from='osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:51:59.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:58 smithi110 ceph-mon[30535]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:51:59.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:59.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:51:59.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:58 smithi110 ceph-mon[30535]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:52:00.067 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[25338]: purged_snaps scrub starts 2024-03-11T21:52:00.067 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[25338]: purged_snaps scrub ok 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[25338]: osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643] boot 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[25338]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[34177]: purged_snaps scrub starts 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[34177]: purged_snaps scrub ok 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[34177]: osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643] boot 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[34177]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:52:00.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:51:59 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:52:00.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:59 smithi110 ceph-mon[30535]: purged_snaps scrub starts 2024-03-11T21:52:00.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:59 smithi110 ceph-mon[30535]: purged_snaps scrub ok 2024-03-11T21:52:00.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:52:00.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:59 smithi110 ceph-mon[30535]: osd.1 [v2:172.21.15.102:6810/265722643,v1:172.21.15.102:6811/265722643] boot 2024-03-11T21:52:00.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:59 smithi110 ceph-mon[30535]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:52:00.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:51:59 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:52:01.015 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/nvme2n1 2024-03-11T21:52:01.015 INFO:teuthology.orchestra.run.smithi102.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:52:01.015 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-11T21:52:01.015 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-11T21:52:01.015 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-11T21:52:01.015 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128835 s, 814 MB/s 2024-03-11T21:52:01.015 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-11T21:52:01.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:00 smithi102 ceph-mon[25338]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:52:01.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:00 smithi102 ceph-mon[25338]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:01.075 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:00 smithi102 ceph-mon[34177]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:52:01.075 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:00 smithi102 ceph-mon[34177]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:01.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:00 smithi110 ceph-mon[30535]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:52:01.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:00 smithi110 ceph-mon[30535]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:01.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi102:/dev/nvme2n1 2024-03-11T21:52:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:52:02.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: Detected new or changed devices on smithi102 2024-03-11T21:52:02.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:02.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi102 to 8032M 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[34177]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: Detected new or changed devices on smithi102 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi102 to 8032M 2024-03-11T21:52:02.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.660 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:02.660 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:02.660 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.660 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:02 smithi102 ceph-mon[25338]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: Detected new or changed devices on smithi102 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi102 to 8032M 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:02.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:02.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:02 smithi110 ceph-mon[30535]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:04.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:03 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:52:04.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:03 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:52:04.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:03 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:04.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:03 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:52:04.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:03 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:52:04.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:03 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:03 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:52:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:03 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:52:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:03 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:04 smithi102 ceph-mon[25338]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:52:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:04 smithi102 ceph-mon[25338]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:05.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:04 smithi102 ceph-mon[34177]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:52:05.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:04 smithi102 ceph-mon[34177]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:05.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:04 smithi110 ceph-mon[30535]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:52:05.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:04 smithi110 ceph-mon[30535]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[25338]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/301881262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c25d672-a2ee-420d-9008-cb48d85c3b57"}]: dispatch 2024-03-11T21:52:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c25d672-a2ee-420d-9008-cb48d85c3b57"}]: dispatch 2024-03-11T21:52:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c25d672-a2ee-420d-9008-cb48d85c3b57"}]': finished 2024-03-11T21:52:07.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[25338]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:52:07.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[34177]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/301881262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c25d672-a2ee-420d-9008-cb48d85c3b57"}]: dispatch 2024-03-11T21:52:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c25d672-a2ee-420d-9008-cb48d85c3b57"}]: dispatch 2024-03-11T21:52:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c25d672-a2ee-420d-9008-cb48d85c3b57"}]': finished 2024-03-11T21:52:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[34177]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:52:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:07.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:07 smithi110 ceph-mon[30535]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:07 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/301881262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c25d672-a2ee-420d-9008-cb48d85c3b57"}]: dispatch 2024-03-11T21:52:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:07 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c25d672-a2ee-420d-9008-cb48d85c3b57"}]: dispatch 2024-03-11T21:52:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:07 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c25d672-a2ee-420d-9008-cb48d85c3b57"}]': finished 2024-03-11T21:52:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:07 smithi110 ceph-mon[30535]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:52:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:08.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:08 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/3229161364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:52:08.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:08 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/3229161364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:52:08.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:08 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/3229161364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:52:09.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:09 smithi102 ceph-mon[25338]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:09.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:09 smithi102 ceph-mon[34177]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:09.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:09 smithi110 ceph-mon[30535]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:11.554 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:11 smithi102 ceph-mon[25338]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:11.555 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:11 smithi102 ceph-mon[34177]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:11.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:11 smithi110 ceph-mon[30535]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:13.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:13 smithi102 ceph-mon[25338]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:13.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:13 smithi102 ceph-mon[34177]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:13.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:13 smithi110 ceph-mon[30535]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:14.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:14 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:52:14.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:14 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:14.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:14 smithi102 ceph-mon[25338]: Deploying daemon osd.2 on smithi102 2024-03-11T21:52:14.639 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:14 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:52:14.639 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:14 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:14.640 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:14 smithi102 ceph-mon[34177]: Deploying daemon osd.2 on smithi102 2024-03-11T21:52:14.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:14 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:52:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:14 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:14 smithi110 ceph-mon[30535]: Deploying daemon osd.2 on smithi102 2024-03-11T21:52:15.475 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:15 smithi102 ceph-mon[25338]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:15.475 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:15 smithi102 ceph-mon[34177]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:15.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:15 smithi110 ceph-mon[30535]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:17 smithi102 ceph-mon[25338]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:17.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:17 smithi102 ceph-mon[34177]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:17.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:17 smithi110 ceph-mon[30535]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:19.542 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:19 smithi102 ceph-mon[34177]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:19.543 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:19 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:19.543 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:19 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:19.543 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:19 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:19.543 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:19 smithi102 ceph-mon[25338]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:19.543 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:19 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:19.543 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:19 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:19.543 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:19 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:19.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:19 smithi110 ceph-mon[30535]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:19.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:19 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:19.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:19 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:19.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:19 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:21.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:21 smithi102 ceph-mon[25338]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:21.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:21 smithi102 ceph-mon[34177]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:21.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:21 smithi110 ceph-mon[30535]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:22.033 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 2 on host 'smithi102' 2024-03-11T21:52:22.638 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:22 smithi102 ceph-mon[25338]: from='osd.2 [v2:172.21.15.102:6818/2501373355,v1:172.21.15.102:6819/2501373355]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:52:22.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:22 smithi102 ceph-mon[25338]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:52:22.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:22 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:22.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:22 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:22.639 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:22 smithi102 ceph-mon[34177]: from='osd.2 [v2:172.21.15.102:6818/2501373355,v1:172.21.15.102:6819/2501373355]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:52:22.639 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:22 smithi102 ceph-mon[34177]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:52:22.639 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:22 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:22.639 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:22 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:22.662 DEBUG:teuthology.orchestra.run.smithi102:osd.2> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T21:52:22.664 INFO:tasks.cephadm:Deploying osd.3 on smithi102 with /dev/nvme1n1... 2024-03-11T21:52:22.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-11T21:52:22.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:22 smithi110 ceph-mon[30535]: from='osd.2 [v2:172.21.15.102:6818/2501373355,v1:172.21.15.102:6819/2501373355]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:52:22.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:22 smithi110 ceph-mon[30535]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:52:22.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:22 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:22.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:22 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:22.960 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:52:23.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[34177]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:23.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[34177]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:52:23.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[34177]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:52:23.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:23.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[34177]: from='osd.2 [v2:172.21.15.102:6818/2501373355,v1:172.21.15.102:6819/2501373355]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:23.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[34177]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:23.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:23.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:23.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2[53625]: 2024-03-11T21:52:23.410+0000 7fb0ec5e3700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T21:52:23.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2[53625]: 2024-03-11T21:52:23.413+0000 7fb0e83d6700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:52:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[25338]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[25338]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:52:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[25338]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:52:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[25338]: from='osd.2 [v2:172.21.15.102:6818/2501373355,v1:172.21.15.102:6819/2501373355]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[25338]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:23 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:23.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:23 smithi110 ceph-mon[30535]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:23.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:23 smithi110 ceph-mon[30535]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:52:23.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:23 smithi110 ceph-mon[30535]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:52:23.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:23 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:23.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:23 smithi110 ceph-mon[30535]: from='osd.2 [v2:172.21.15.102:6818/2501373355,v1:172.21.15.102:6819/2501373355]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:23.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:23 smithi110 ceph-mon[30535]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:23.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:23 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:23.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:23 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:24.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:24 smithi102 ceph-mon[25338]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:52:24.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:24 smithi102 ceph-mon[25338]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:52:24.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:24.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:24.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:24 smithi102 ceph-mon[34177]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:52:24.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:24 smithi102 ceph-mon[34177]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:52:24.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:24 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:24.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:24 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:24.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:24 smithi110 ceph-mon[30535]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:52:24.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:24 smithi110 ceph-mon[30535]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:52:24.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:24 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:24.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:24 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:25.449 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/nvme1n1 2024-03-11T21:52:25.449 INFO:teuthology.orchestra.run.smithi102.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:52:25.449 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-11T21:52:25.449 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-11T21:52:25.449 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-11T21:52:25.449 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144931 s, 723 MB/s 2024-03-11T21:52:25.449 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-11T21:52:25.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[34177]: purged_snaps scrub starts 2024-03-11T21:52:25.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[34177]: purged_snaps scrub ok 2024-03-11T21:52:25.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[34177]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:25.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:25.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[34177]: osd.2 [v2:172.21.15.102:6818/2501373355,v1:172.21.15.102:6819/2501373355] boot 2024-03-11T21:52:25.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[34177]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:52:25.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[25338]: purged_snaps scrub starts 2024-03-11T21:52:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[25338]: purged_snaps scrub ok 2024-03-11T21:52:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[25338]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[25338]: osd.2 [v2:172.21.15.102:6818/2501373355,v1:172.21.15.102:6819/2501373355] boot 2024-03-11T21:52:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[25338]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:52:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:25 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:25.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:25 smithi110 ceph-mon[30535]: purged_snaps scrub starts 2024-03-11T21:52:25.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:25 smithi110 ceph-mon[30535]: purged_snaps scrub ok 2024-03-11T21:52:25.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:25 smithi110 ceph-mon[30535]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:52:25.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:25 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:25.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:25 smithi110 ceph-mon[30535]: osd.2 [v2:172.21.15.102:6818/2501373355,v1:172.21.15.102:6819/2501373355] boot 2024-03-11T21:52:25.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:25 smithi110 ceph-mon[30535]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:52:25.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:25 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:52:26.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi102:/dev/nvme1n1 2024-03-11T21:52:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:26 smithi102 ceph-mon[25338]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:52:26.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:26 smithi102 ceph-mon[34177]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:52:26.846 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:52:26.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:26 smithi110 ceph-mon[30535]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:52:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[25338]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:27.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:52:27.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:52:27.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[25338]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:52:27.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:52:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[34177]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:52:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:52:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[34177]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:52:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:27 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:52:27.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:27 smithi110 ceph-mon[30535]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:27.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:27 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:52:27.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:27 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:52:27.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:27 smithi110 ceph-mon[30535]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:52:27.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:27 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:52:28.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60289]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:52:28.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60289]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:28.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60289]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:28.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60289]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:28.870 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:28 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:52:28.870 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:28 smithi102 ceph-mon[25338]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:52:28.870 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:28 smithi102 ceph-mon[25338]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:28.870 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:28 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:28.871 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60292]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:52:28.871 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60292]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:28.871 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60292]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:28.871 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60292]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:28.871 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:28 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:52:28.872 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:28 smithi102 ceph-mon[34177]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:52:28.872 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:28 smithi102 ceph-mon[34177]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:28.872 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:28 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:28.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:28 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:52:28.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:28 smithi110 ceph-mon[30535]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:52:28.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:28 smithi110 ceph-mon[30535]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:28.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:28 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.129 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60331]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme3n1 2024-03-11T21:52:29.129 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60331]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:29.130 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60331]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:29.130 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:52:28 smithi102 sudo[60331]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:29.410 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60373]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme3n1 2024-03-11T21:52:29.410 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60373]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:29.410 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60373]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:29.410 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60373]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:29.668 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60431]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2024-03-11T21:52:29.668 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60431]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:29.668 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60431]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:29.668 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60431]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:29.950 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60465]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme2n1 2024-03-11T21:52:29.950 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60465]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:29.951 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60465]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:29.951 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60465]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:29.951 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:52:29.951 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: Detected new or changed devices on smithi102 2024-03-11T21:52:29.951 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.951 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.951 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:29.951 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:29.951 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi102 to 5354M 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:29.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60524]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: Detected new or changed devices on smithi102 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi102 to 5354M 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:29.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:52:29.954 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:29 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:29.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: Detected new or changed devices on smithi102 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi102 to 5354M 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:29.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:29.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:52:29.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:52:29.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:52:29.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:29 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:30.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60524]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:30.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60524]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:30.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:29 smithi102 sudo[60524]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 sudo[60544]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:52:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 sudo[60544]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 sudo[60544]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 sudo[60544]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:30.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 sudo[33116]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:52:30.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 sudo[33116]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:30.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 sudo[33116]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:30.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 sudo[33116]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:30.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 sudo[33119]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 sudo[33119]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 sudo[33119]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 sudo[33119]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:52:30.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:52:30.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:52:30.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:30 smithi110 ceph-mon[30535]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:30.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:30.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:52:30.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:52:30.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:52:30.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:52:30.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:52:30.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T21:52:30.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[25338]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:30.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:52:30.994 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:52:30.994 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:52:30.994 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:30 smithi102 ceph-mon[34177]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:31.270 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 sudo[60571]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:52:31.273 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 sudo[60571]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:31.273 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 sudo[60571]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:31.273 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 sudo[60571]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:31.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 sudo[60578]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:52:31.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 sudo[60578]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:52:31.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 sudo[60578]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:52:31.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 sudo[60578]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:52:31.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:31 smithi110 ceph-mon[30535]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:52:31.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:31 smithi110 ceph-mon[30535]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:52:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:31 smithi110 ceph-mon[30535]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:52:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:52:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:52:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:52:31.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:31 smithi110 ceph-mon[30535]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:52:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[34177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:52:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[34177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:52:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[34177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:52:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:52:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:52:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:52:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[34177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:52:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[25338]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:52:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[25338]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:52:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[25338]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:52:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:52:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:52:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:52:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:31 smithi102 ceph-mon[25338]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:52:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2124668062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89807137-de78-4a04-8442-478a3e2eb242"}]: dispatch 2024-03-11T21:52:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89807137-de78-4a04-8442-478a3e2eb242"}]: dispatch 2024-03-11T21:52:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89807137-de78-4a04-8442-478a3e2eb242"}]': finished 2024-03-11T21:52:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[25338]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T21:52:33.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:33.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[25338]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/2124668062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89807137-de78-4a04-8442-478a3e2eb242"}]: dispatch 2024-03-11T21:52:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89807137-de78-4a04-8442-478a3e2eb242"}]: dispatch 2024-03-11T21:52:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89807137-de78-4a04-8442-478a3e2eb242"}]': finished 2024-03-11T21:52:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[34177]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T21:52:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:32 smithi102 ceph-mon[34177]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:33.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:32 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/2124668062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89807137-de78-4a04-8442-478a3e2eb242"}]: dispatch 2024-03-11T21:52:33.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:32 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89807137-de78-4a04-8442-478a3e2eb242"}]: dispatch 2024-03-11T21:52:33.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:32 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89807137-de78-4a04-8442-478a3e2eb242"}]': finished 2024-03-11T21:52:33.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:32 smithi110 ceph-mon[30535]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T21:52:33.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:32 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:33.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:32 smithi110 ceph-mon[30535]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:34 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/316818004' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:52:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:34 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/316818004' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:52:34.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:34 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/316818004' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:52:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:35 smithi102 ceph-mon[25338]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:35.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:35 smithi102 ceph-mon[34177]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:35.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:35 smithi110 ceph-mon[30535]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:37.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:37 smithi102 ceph-mon[25338]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T21:52:37.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:37 smithi102 ceph-mon[34177]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T21:52:37.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:37 smithi110 ceph-mon[30535]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T21:52:39.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:39 smithi102 ceph-mon[25338]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-03-11T21:52:39.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:39 smithi102 ceph-mon[34177]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-03-11T21:52:39.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:39 smithi110 ceph-mon[30535]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-03-11T21:52:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T21:52:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:40 smithi102 ceph-mon[25338]: Deploying daemon osd.3 on smithi102 2024-03-11T21:52:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:40 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T21:52:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:40 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:40 smithi102 ceph-mon[34177]: Deploying daemon osd.3 on smithi102 2024-03-11T21:52:40.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:40 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T21:52:40.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:40 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:40.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:40 smithi110 ceph-mon[30535]: Deploying daemon osd.3 on smithi102 2024-03-11T21:52:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:41 smithi102 ceph-mon[25338]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-11T21:52:41.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:41 smithi102 ceph-mon[34177]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-11T21:52:41.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:41 smithi110 ceph-mon[30535]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-11T21:52:43.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:43 smithi102 ceph-mon[25338]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 128 KiB/s wr, 8 op/s 2024-03-11T21:52:43.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:43 smithi102 ceph-mon[34177]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 128 KiB/s wr, 8 op/s 2024-03-11T21:52:43.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:43 smithi110 ceph-mon[30535]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 128 KiB/s wr, 8 op/s 2024-03-11T21:52:45.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:45 smithi102 ceph-mon[25338]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-11T21:52:45.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:45.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:45.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:45 smithi102 ceph-mon[34177]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-11T21:52:45.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:45.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:45.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:45.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:45 smithi110 ceph-mon[30535]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-11T21:52:45.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:45.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:45.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:47.647 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:47 smithi102 ceph-mon[25338]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-11T21:52:47.648 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:47 smithi102 ceph-mon[34177]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-11T21:52:47.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:47 smithi110 ceph-mon[30535]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-11T21:52:49.036 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 3 on host 'smithi102' 2024-03-11T21:52:49.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:49.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:49.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:48 smithi110 ceph-mon[30535]: from='osd.3 [v2:172.21.15.102:6826/1356933888,v1:172.21.15.102:6827/1356933888]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:52:49.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:48 smithi110 ceph-mon[30535]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:52:49.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:48 smithi110 ceph-mon[30535]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:49.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:49.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:49.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:49.347 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:49.347 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[25338]: from='osd.3 [v2:172.21.15.102:6826/1356933888,v1:172.21.15.102:6827/1356933888]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[25338]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[25338]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[34177]: from='osd.3 [v2:172.21.15.102:6826/1356933888,v1:172.21.15.102:6827/1356933888]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[34177]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[34177]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:49.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:49.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:49.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:49.649 DEBUG:teuthology.orchestra.run.smithi102:osd.3> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T21:52:49.651 INFO:tasks.cephadm:Deploying osd.4 on smithi110 with /dev/nvme4n1... 2024-03-11T21:52:49.651 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:52:49.894 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:52:50.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:49 smithi110 ceph-mon[30535]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:52:50.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:49 smithi110 ceph-mon[30535]: from='osd.3 [v2:172.21.15.102:6826/1356933888,v1:172.21.15.102:6827/1356933888]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:50.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:49 smithi110 ceph-mon[30535]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T21:52:50.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:50.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:49 smithi110 ceph-mon[30535]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:50.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:50.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:50.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[25338]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:52:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[25338]: from='osd.3 [v2:172.21.15.102:6826/1356933888,v1:172.21.15.102:6827/1356933888]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[25338]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T21:52:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[25338]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[34177]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[34177]: from='osd.3 [v2:172.21.15.102:6826/1356933888,v1:172.21.15.102:6827/1356933888]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[34177]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[34177]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:52:50.796 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3[62365]: 2024-03-11T21:52:50.450+0000 7f36297ec700 -1 osd.3 0 waiting for initial osdmap 2024-03-11T21:52:50.797 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3[62365]: 2024-03-11T21:52:50.458+0000 7f36215d7700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:52:51.084 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[34177]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[34177]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[34177]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[34177]: from='osd.3 ' entity='osd.3' 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[25338]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[25338]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:51.085 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[25338]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:51.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:50 smithi102 ceph-mon[25338]: from='osd.3 ' entity='osd.3' 2024-03-11T21:52:51.196 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:50 smithi110 ceph-mon[30535]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-11T21:52:51.196 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:50 smithi110 ceph-mon[30535]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T21:52:51.196 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:50 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:51.197 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:50 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:51.197 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:50 smithi110 ceph-mon[30535]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:52:51.197 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:50 smithi110 ceph-mon[30535]: from='osd.3 ' entity='osd.3' 2024-03-11T21:52:51.744 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:52:51.744 INFO:teuthology.orchestra.run.smithi110.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:52:51.744 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:52:51.744 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-03-11T21:52:51.744 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-03-11T21:52:51.744 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00956456 s, 1.1 GB/s 2024-03-11T21:52:51.744 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-03-11T21:52:52.081 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:51 smithi110 ceph-mon[30535]: purged_snaps scrub starts 2024-03-11T21:52:52.081 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:51 smithi110 ceph-mon[30535]: purged_snaps scrub ok 2024-03-11T21:52:52.081 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:51 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:52.081 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:51 smithi110 ceph-mon[30535]: osd.3 [v2:172.21.15.102:6826/1356933888,v1:172.21.15.102:6827/1356933888] boot 2024-03-11T21:52:52.081 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:51 smithi110 ceph-mon[30535]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T21:52:52.081 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:51 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:52.366 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi110:/dev/nvme4n1 2024-03-11T21:52:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[25338]: purged_snaps scrub starts 2024-03-11T21:52:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[25338]: purged_snaps scrub ok 2024-03-11T21:52:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[25338]: osd.3 [v2:172.21.15.102:6826/1356933888,v1:172.21.15.102:6827/1356933888] boot 2024-03-11T21:52:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[25338]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T21:52:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[34177]: purged_snaps scrub starts 2024-03-11T21:52:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[34177]: purged_snaps scrub ok 2024-03-11T21:52:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[34177]: osd.3 [v2:172.21.15.102:6826/1356933888,v1:172.21.15.102:6827/1356933888] boot 2024-03-11T21:52:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[34177]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T21:52:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:51 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:52:52.604 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:52:53.155 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:52 smithi102 ceph-mon[25338]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:53.155 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:52 smithi102 ceph-mon[34177]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:53.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:52 smithi110 ceph-mon[30535]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:54.225 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T21:52:54.225 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: Detected new or changed devices on smithi102 2024-03-11T21:52:54.225 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.225 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.225 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.226 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.226 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.226 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.226 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi102 to 4016M 2024-03-11T21:52:54.226 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.226 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:54.226 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:54.226 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:53 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: Detected new or changed devices on smithi102 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi102 to 4016M 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: Detected new or changed devices on smithi102 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi102 to 4016M 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:52:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:53 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:52:55.252 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:55 smithi110 ceph-mon[30535]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:55.252 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:55 smithi110 ceph-mon[30535]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:52:55.252 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:55 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:52:55.253 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:55 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:52:55.253 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:55 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[25338]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[25338]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:52:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:52:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:52:55.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[34177]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[34177]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:52:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:52:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:52:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:55 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:52:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[34177]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/1139978065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c670cc8e-51e7-4b4c-8e20-160b54ab7688"}]: dispatch 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c670cc8e-51e7-4b4c-8e20-160b54ab7688"}]: dispatch 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c670cc8e-51e7-4b4c-8e20-160b54ab7688"}]': finished 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[34177]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[25338]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/1139978065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c670cc8e-51e7-4b4c-8e20-160b54ab7688"}]: dispatch 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c670cc8e-51e7-4b4c-8e20-160b54ab7688"}]: dispatch 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c670cc8e-51e7-4b4c-8e20-160b54ab7688"}]': finished 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[25338]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T21:52:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:57 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:52:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:57 smithi110 ceph-mon[30535]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:57 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/1139978065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c670cc8e-51e7-4b4c-8e20-160b54ab7688"}]: dispatch 2024-03-11T21:52:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:57 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c670cc8e-51e7-4b4c-8e20-160b54ab7688"}]: dispatch 2024-03-11T21:52:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:57 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c670cc8e-51e7-4b4c-8e20-160b54ab7688"}]': finished 2024-03-11T21:52:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:57 smithi110 ceph-mon[30535]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T21:52:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:57 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:52:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:58 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/2529902447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:52:58.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:58 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/2529902447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:52:58.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:58 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/2529902447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:52:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:52:59 smithi102 ceph-mon[25338]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:59.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:52:59 smithi102 ceph-mon[34177]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:52:59.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:52:59 smithi110 ceph-mon[30535]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:01.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:01 smithi102 ceph-mon[25338]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:01.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:01 smithi102 ceph-mon[34177]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:01.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:01 smithi110 ceph-mon[30535]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:03 smithi102 ceph-mon[25338]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:03.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:03 smithi102 ceph-mon[34177]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:03.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:03 smithi110 ceph-mon[30535]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:04 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T21:53:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:04 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:04 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T21:53:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:04 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:04.722 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:04 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T21:53:04.722 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:04 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:05.571 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:05 smithi110 ceph-mon[30535]: Deploying daemon osd.4 on smithi110 2024-03-11T21:53:05.572 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:05 smithi110 ceph-mon[30535]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:05 smithi102 ceph-mon[25338]: Deploying daemon osd.4 on smithi110 2024-03-11T21:53:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:05 smithi102 ceph-mon[25338]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:05 smithi102 ceph-mon[34177]: Deploying daemon osd.4 on smithi110 2024-03-11T21:53:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:05 smithi102 ceph-mon[34177]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:07 smithi102 ceph-mon[25338]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:07 smithi102 ceph-mon[34177]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:07.721 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:07 smithi110 ceph-mon[30535]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:09.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:09 smithi102 ceph-mon[25338]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:09.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:09 smithi102 ceph-mon[34177]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:09.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:09 smithi110 ceph-mon[30535]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:10.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:10 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:10.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:10 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:10.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:10 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:10.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:10 smithi110 ceph-mon[30535]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:10.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:10 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:10.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:10 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:10.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:10 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:10.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:10 smithi102 ceph-mon[25338]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:10.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:10 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:10.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:10 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:10.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:10 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:10.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:10 smithi102 ceph-mon[34177]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:12 smithi110 ceph-mon[30535]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:12 smithi110 ceph-mon[30535]: from='osd.4 [v2:172.21.15.110:6800/882287460,v1:172.21.15.110:6801/882287460]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:53:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:12 smithi110 ceph-mon[30535]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:53:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[25338]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[25338]: from='osd.4 [v2:172.21.15.110:6800/882287460,v1:172.21.15.110:6801/882287460]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[25338]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[34177]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:13.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:13.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[34177]: from='osd.4 [v2:172.21.15.110:6800/882287460,v1:172.21.15.110:6801/882287460]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:53:13.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:12 smithi102 ceph-mon[34177]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:53:13.374 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 4 on host 'smithi110' 2024-03-11T21:53:13.976 DEBUG:teuthology.orchestra.run.smithi110:osd.4> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T21:53:13.978 INFO:tasks.cephadm:Deploying osd.5 on smithi110 with /dev/nvme3n1... 2024-03-11T21:53:13.979 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-11T21:53:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[25338]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[25338]: from='osd.4 [v2:172.21.15.110:6800/882287460,v1:172.21.15.110:6801/882287460]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[25338]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[25338]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[34177]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:53:14.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[34177]: from='osd.4 [v2:172.21.15.110:6800/882287460,v1:172.21.15.110:6801/882287460]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:14.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[34177]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T21:53:14.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:14.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:14 smithi102 ceph-mon[34177]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-mon[30535]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:53:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-mon[30535]: from='osd.4 [v2:172.21.15.110:6800/882287460,v1:172.21.15.110:6801/882287460]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-mon[30535]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T21:53:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-mon[30535]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:14.738 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:53:14.492+0000 7f3799329700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T21:53:14.739 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:53:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:53:14.496+0000 7f379511c700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:53:15.197 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:53:15.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:15 smithi110 ceph-mon[30535]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:15.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:15 smithi110 ceph-mon[30535]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:53:15.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:15 smithi110 ceph-mon[30535]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T21:53:15.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:15 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:15.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:15 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:15.817 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[25338]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:15.817 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[25338]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:53:15.817 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[25338]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T21:53:15.817 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:15.817 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:15.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[34177]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:15.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[34177]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:53:15.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[34177]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T21:53:15.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:15.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:15 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:16.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:16 smithi110 ceph-mon[30535]: purged_snaps scrub starts 2024-03-11T21:53:16.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:16 smithi110 ceph-mon[30535]: purged_snaps scrub ok 2024-03-11T21:53:16.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:16 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:16.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:16 smithi110 ceph-mon[30535]: osd.4 [v2:172.21.15.110:6800/882287460,v1:172.21.15.110:6801/882287460] boot 2024-03-11T21:53:16.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:16 smithi110 ceph-mon[30535]: osdmap e31: 5 total, 5 up, 5 in 2024-03-11T21:53:16.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:16 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:16.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[25338]: purged_snaps scrub starts 2024-03-11T21:53:16.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[25338]: purged_snaps scrub ok 2024-03-11T21:53:16.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:16.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[25338]: osd.4 [v2:172.21.15.110:6800/882287460,v1:172.21.15.110:6801/882287460] boot 2024-03-11T21:53:16.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[25338]: osdmap e31: 5 total, 5 up, 5 in 2024-03-11T21:53:16.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:16.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[34177]: purged_snaps scrub starts 2024-03-11T21:53:16.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[34177]: purged_snaps scrub ok 2024-03-11T21:53:16.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:16.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[34177]: osd.4 [v2:172.21.15.110:6800/882287460,v1:172.21.15.110:6801/882287460] boot 2024-03-11T21:53:16.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[34177]: osdmap e31: 5 total, 5 up, 5 in 2024-03-11T21:53:16.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:16 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:53:17.540 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:17 smithi110 ceph-mon[30535]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:17.541 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:17 smithi110 ceph-mon[30535]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T21:53:17.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:17 smithi102 ceph-mon[25338]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:17.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:17 smithi102 ceph-mon[25338]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T21:53:17.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:17 smithi102 ceph-mon[34177]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:53:17.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:17 smithi102 ceph-mon[34177]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T21:53:18.395 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/nvme3n1 2024-03-11T21:53:18.395 INFO:teuthology.orchestra.run.smithi110.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:53:18.395 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-11T21:53:18.396 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-03-11T21:53:18.396 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-03-11T21:53:18.396 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00955829 s, 1.1 GB/s 2024-03-11T21:53:18.396 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-03-11T21:53:18.730 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:18 smithi110 ceph-mon[30535]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T21:53:18.730 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:18 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.730 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:18 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.730 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:18 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:53:18.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:18 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:18 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:18.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:18 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:18.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:18 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[25338]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T21:53:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:53:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[34177]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T21:53:18.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:53:18.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:18.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:18.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:18.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:18 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:19.063 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi110:/dev/nvme3n1 2024-03-11T21:53:19.302 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:53:19.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:19 smithi110 ceph-mon[30535]: Detected new or changed devices on smithi110 2024-03-11T21:53:19.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:19 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi110 to 17088M 2024-03-11T21:53:19.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:19 smithi110 ceph-mon[30535]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:19.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:19 smithi110 ceph-mon[30535]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T21:53:19.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:19 smithi102 ceph-mon[25338]: Detected new or changed devices on smithi110 2024-03-11T21:53:19.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:19 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi110 to 17088M 2024-03-11T21:53:19.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:19 smithi102 ceph-mon[25338]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:19.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:19 smithi102 ceph-mon[25338]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T21:53:19.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:19 smithi102 ceph-mon[34177]: Detected new or changed devices on smithi110 2024-03-11T21:53:19.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:19 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi110 to 17088M 2024-03-11T21:53:19.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:19 smithi102 ceph-mon[34177]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:19.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:19 smithi102 ceph-mon[34177]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T21:53:20.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:20 smithi110 ceph-mon[30535]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T21:53:20.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:20 smithi102 ceph-mon[25338]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T21:53:20.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:20 smithi102 ceph-mon[34177]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T21:53:21.860 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:21 smithi110 ceph-mon[30535]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T21:53:21.860 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:21 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:53:21.860 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:21 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:53:21.860 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:21 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:21.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:21 smithi102 ceph-mon[25338]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T21:53:21.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:21 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:53:21.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:21 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:53:21.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:21 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:21.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:21 smithi102 ceph-mon[34177]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T21:53:21.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:21 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:53:21.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:21 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:53:21.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:21 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:22.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:22 smithi110 ceph-mon[30535]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:53:22.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:22 smithi102 ceph-mon[25338]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:53:22.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:22 smithi102 ceph-mon[34177]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:53:23.895 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:23 smithi110 ceph-mon[30535]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-11T21:53:23.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:23 smithi102 ceph-mon[25338]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-11T21:53:23.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:23 smithi102 ceph-mon[34177]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-11T21:53:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/53154983' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e087753a-eddf-4880-a51e-8efdfec06331"}]: dispatch 2024-03-11T21:53:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e087753a-eddf-4880-a51e-8efdfec06331"}]: dispatch 2024-03-11T21:53:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e087753a-eddf-4880-a51e-8efdfec06331"}]': finished 2024-03-11T21:53:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[25338]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T21:53:24.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:24.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/53154983' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e087753a-eddf-4880-a51e-8efdfec06331"}]: dispatch 2024-03-11T21:53:24.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e087753a-eddf-4880-a51e-8efdfec06331"}]: dispatch 2024-03-11T21:53:24.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e087753a-eddf-4880-a51e-8efdfec06331"}]': finished 2024-03-11T21:53:24.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[34177]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T21:53:24.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:24 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:24.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:24 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/53154983' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e087753a-eddf-4880-a51e-8efdfec06331"}]: dispatch 2024-03-11T21:53:24.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:24 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e087753a-eddf-4880-a51e-8efdfec06331"}]: dispatch 2024-03-11T21:53:24.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:24 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e087753a-eddf-4880-a51e-8efdfec06331"}]': finished 2024-03-11T21:53:24.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:24 smithi110 ceph-mon[30535]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T21:53:24.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:24 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:25 smithi102 ceph-mon[25338]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:53:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:25 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/2830951146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:53:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:25 smithi102 ceph-mon[34177]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:53:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:25 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/2830951146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:53:25.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:25 smithi110 ceph-mon[30535]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:53:25.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:25 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/2830951146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:53:27.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:27 smithi110 ceph-mon[30535]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-11T21:53:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:27 smithi102 ceph-mon[25338]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-11T21:53:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:27 smithi102 ceph-mon[34177]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-11T21:53:29.632 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:29 smithi110 ceph-mon[30535]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-11T21:53:29.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:29 smithi102 ceph-mon[25338]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-11T21:53:29.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:29 smithi102 ceph-mon[34177]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-11T21:53:31.884 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:31 smithi110 ceph-mon[30535]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:31.884 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T21:53:31.884 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:31.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:31 smithi102 ceph-mon[25338]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:31.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T21:53:31.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:31.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:31 smithi102 ceph-mon[34177]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:31.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T21:53:31.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:32.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:32 smithi110 ceph-mon[30535]: Deploying daemon osd.5 on smithi110 2024-03-11T21:53:32.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:32 smithi102 ceph-mon[34177]: Deploying daemon osd.5 on smithi110 2024-03-11T21:53:32.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:32 smithi102 ceph-mon[25338]: Deploying daemon osd.5 on smithi110 2024-03-11T21:53:33.915 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:33 smithi110 ceph-mon[30535]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:33 smithi102 ceph-mon[25338]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:33.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:33 smithi102 ceph-mon[34177]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:35.807 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:35 smithi110 ceph-mon[30535]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:35 smithi102 ceph-mon[25338]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:35 smithi102 ceph-mon[34177]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:37.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:37 smithi110 ceph-mon[30535]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:37.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:37.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:37.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:37 smithi102 ceph-mon[25338]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:37.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:37 smithi102 ceph-mon[34177]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:37.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:37.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:37.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:38.699 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:38 smithi110 ceph-mon[30535]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:38 smithi102 ceph-mon[34177]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:38.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:38 smithi102 ceph-mon[25338]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:40.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:40 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:40.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:40 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:40.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:40 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:40.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:40 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:40.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:40 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:40.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:40 smithi110 ceph-mon[30535]: from='osd.5 [v2:172.21.15.110:6808/1539634603,v1:172.21.15.110:6809/1539634603]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:53:40.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:40 smithi110 ceph-mon[30535]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:53:40.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:40 smithi110 ceph-mon[30535]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:40.817 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 5 on host 'smithi110' 2024-03-11T21:53:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[25338]: from='osd.5 [v2:172.21.15.110:6808/1539634603,v1:172.21.15.110:6809/1539634603]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[25338]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[25338]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:41.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:41.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[34177]: from='osd.5 [v2:172.21.15.110:6808/1539634603,v1:172.21.15.110:6809/1539634603]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:53:41.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[34177]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:53:41.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:40 smithi102 ceph-mon[34177]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:41.489 DEBUG:teuthology.orchestra.run.smithi110:osd.5> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T21:53:41.491 INFO:tasks.cephadm:Deploying osd.6 on smithi110 with /dev/nvme2n1... 2024-03-11T21:53:41.491 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-11T21:53:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:41 smithi110 ceph-mon[30535]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T21:53:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:41 smithi110 ceph-mon[30535]: from='osd.5 [v2:172.21.15.110:6808/1539634603,v1:172.21.15.110:6809/1539634603]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:41 smithi110 ceph-mon[30535]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T21:53:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:41 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:41 smithi110 ceph-mon[30535]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:41 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:41 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:41.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:41 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:42.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[25338]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[25338]: from='osd.5 [v2:172.21.15.110:6808/1539634603,v1:172.21.15.110:6809/1539634603]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[25338]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[25338]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[34177]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[34177]: from='osd.5 [v2:172.21.15.110:6808/1539634603,v1:172.21.15.110:6809/1539634603]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[34177]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[34177]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:53:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:42.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:42.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:41 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:53:42.238 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:53:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:53:42.000+0000 7f164db8b700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T21:53:42.238 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:53:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:53:42.004+0000 7f1645976700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:53:42.668 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:53:42.929 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:42 smithi110 ceph-mon[30535]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:53:42.929 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:42 smithi110 ceph-mon[30535]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T21:53:42.929 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:42 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:42.929 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:42 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:42.929 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:42 smithi110 ceph-mon[30535]: from='osd.5 ' entity='osd.5' 2024-03-11T21:53:42.930 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:42 smithi110 ceph-mon[30535]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:43.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[34177]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:53:43.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[34177]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[34177]: from='osd.5 ' entity='osd.5' 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[34177]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[25338]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[25338]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[25338]: from='osd.5 ' entity='osd.5' 2024-03-11T21:53:43.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:42 smithi102 ceph-mon[25338]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:53:43.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:43 smithi110 ceph-mon[30535]: purged_snaps scrub starts 2024-03-11T21:53:43.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:43 smithi110 ceph-mon[30535]: purged_snaps scrub ok 2024-03-11T21:53:43.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:43 smithi110 ceph-mon[30535]: osd.5 [v2:172.21.15.110:6808/1539634603,v1:172.21.15.110:6809/1539634603] boot 2024-03-11T21:53:43.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:43 smithi110 ceph-mon[30535]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T21:53:43.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:43 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:44.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[25338]: purged_snaps scrub starts 2024-03-11T21:53:44.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[25338]: purged_snaps scrub ok 2024-03-11T21:53:44.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[25338]: osd.5 [v2:172.21.15.110:6808/1539634603,v1:172.21.15.110:6809/1539634603] boot 2024-03-11T21:53:44.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[25338]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T21:53:44.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:44.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[34177]: purged_snaps scrub starts 2024-03-11T21:53:44.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[34177]: purged_snaps scrub ok 2024-03-11T21:53:44.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[34177]: osd.5 [v2:172.21.15.110:6808/1539634603,v1:172.21.15.110:6809/1539634603] boot 2024-03-11T21:53:44.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[34177]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T21:53:44.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:43 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:53:44.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:44 smithi110 ceph-mon[30535]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T21:53:44.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:44 smithi110 ceph-mon[30535]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 558 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:45.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:44 smithi102 ceph-mon[25338]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T21:53:45.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:44 smithi102 ceph-mon[25338]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 558 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:45.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:44 smithi102 ceph-mon[34177]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T21:53:45.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:44 smithi102 ceph-mon[34177]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 558 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/nvme2n1 2024-03-11T21:53:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:53:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-11T21:53:45.102 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-03-11T21:53:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-03-11T21:53:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116432 s, 901 MB/s 2024-03-11T21:53:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: Detected new or changed devices on smithi110 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi110 to 8544M 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:45.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:45.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:45.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:45 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:46.097 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi110:/dev/nvme2n1 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: Detected new or changed devices on smithi110 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi110 to 8544M 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: Detected new or changed devices on smithi110 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi110 to 8544M 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:53:46.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:45 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:53:46.336 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:53:46.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:46 smithi110 ceph-mon[30535]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T21:53:46.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:46 smithi110 ceph-mon[30535]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 558 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:46 smithi102 ceph-mon[25338]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T21:53:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:46 smithi102 ceph-mon[25338]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 558 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:46 smithi102 ceph-mon[34177]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T21:53:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:46 smithi102 ceph-mon[34177]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 558 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:47.956 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:47 smithi110 ceph-mon[30535]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T21:53:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:47 smithi102 ceph-mon[25338]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T21:53:48.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:47 smithi102 ceph-mon[34177]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T21:53:49.009 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:48 smithi110 ceph-mon[30535]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:53:49.009 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:53:49.009 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:53:49.009 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:48 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:49.009 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:48 smithi110 ceph-mon[30535]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[25338]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:53:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:53:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:53:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[25338]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[34177]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:53:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:53:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:53:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:48 smithi102 ceph-mon[34177]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:50.024 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:49 smithi110 ceph-mon[30535]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T21:53:50.024 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:49 smithi110 ceph-mon[30535]: Cluster is now healthy 2024-03-11T21:53:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:49 smithi102 ceph-mon[25338]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T21:53:50.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:49 smithi102 ceph-mon[25338]: Cluster is now healthy 2024-03-11T21:53:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:49 smithi102 ceph-mon[34177]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T21:53:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:49 smithi102 ceph-mon[34177]: Cluster is now healthy 2024-03-11T21:53:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:50 smithi102 ceph-mon[25338]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:53:51.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:50 smithi102 ceph-mon[34177]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:53:51.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:50 smithi110 ceph-mon[30535]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/4069851281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e599016e-0eeb-4fdd-a8a9-6fc8eed1218a"}]: dispatch 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e599016e-0eeb-4fdd-a8a9-6fc8eed1218a"}]: dispatch 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e599016e-0eeb-4fdd-a8a9-6fc8eed1218a"}]': finished 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[25338]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/4069851281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e599016e-0eeb-4fdd-a8a9-6fc8eed1218a"}]: dispatch 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e599016e-0eeb-4fdd-a8a9-6fc8eed1218a"}]: dispatch 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e599016e-0eeb-4fdd-a8a9-6fc8eed1218a"}]': finished 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[34177]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T21:53:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:51 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:53:52.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:51 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/4069851281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e599016e-0eeb-4fdd-a8a9-6fc8eed1218a"}]: dispatch 2024-03-11T21:53:52.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:51 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e599016e-0eeb-4fdd-a8a9-6fc8eed1218a"}]: dispatch 2024-03-11T21:53:52.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:51 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e599016e-0eeb-4fdd-a8a9-6fc8eed1218a"}]': finished 2024-03-11T21:53:52.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:51 smithi110 ceph-mon[30535]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T21:53:52.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:51 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:53:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:52 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/3317751142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:53:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:52 smithi102 ceph-mon[25338]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-11T21:53:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:52 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/3317751142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:53:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:52 smithi102 ceph-mon[34177]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-11T21:53:53.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:52 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/3317751142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:53:53.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:52 smithi110 ceph-mon[30535]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-11T21:53:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:55 smithi102 ceph-mon[25338]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T21:53:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:55 smithi102 ceph-mon[34177]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T21:53:55.679 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:55 smithi110 ceph-mon[30535]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T21:53:57.657 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:57 smithi110 ceph-mon[30535]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T21:53:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:57 smithi102 ceph-mon[25338]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T21:53:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:57 smithi102 ceph-mon[34177]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T21:53:58.656 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T21:53:58.656 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T21:53:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:58.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T21:53:58.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:53:59.634 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:59 smithi110 ceph-mon[30535]: Deploying daemon osd.6 on smithi110 2024-03-11T21:53:59.634 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:53:59 smithi110 ceph-mon[30535]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:59 smithi102 ceph-mon[25338]: Deploying daemon osd.6 on smithi110 2024-03-11T21:53:59.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:53:59 smithi102 ceph-mon[25338]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:53:59.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:59 smithi102 ceph-mon[34177]: Deploying daemon osd.6 on smithi110 2024-03-11T21:53:59.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:53:59 smithi102 ceph-mon[34177]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:01.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:01 smithi102 ceph-mon[25338]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:01.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:01 smithi102 ceph-mon[34177]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:01.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:01 smithi110 ceph-mon[30535]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:03.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:03 smithi110 ceph-mon[30535]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:03 smithi102 ceph-mon[25338]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:03.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:03 smithi102 ceph-mon[34177]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:04 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:05.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:04 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:05.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:04 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:05.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:04 smithi102 ceph-mon[25338]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:04 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:04 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:04 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:04 smithi102 ceph-mon[34177]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:05.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:04 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:05.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:04 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:05.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:04 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:05.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:04 smithi110 ceph-mon[30535]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:07.643 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:07 smithi110 ceph-mon[30535]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:07.643 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:07.643 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:07.644 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:07.644 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:07.644 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:07.644 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:07 smithi110 ceph-mon[30535]: from='osd.6 [v2:172.21.15.110:6816/462580487,v1:172.21.15.110:6817/462580487]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:54:07.644 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:07 smithi110 ceph-mon[30535]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:54:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[25338]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[25338]: from='osd.6 [v2:172.21.15.110:6816/462580487,v1:172.21.15.110:6817/462580487]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[25338]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[34177]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[34177]: from='osd.6 [v2:172.21.15.110:6816/462580487,v1:172.21.15.110:6817/462580487]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:54:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:07 smithi102 ceph-mon[34177]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:54:07.988 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 6 on host 'smithi110' 2024-03-11T21:54:08.598 DEBUG:teuthology.orchestra.run.smithi110:osd.6> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T21:54:08.600 INFO:tasks.cephadm:Deploying osd.7 on smithi110 with /dev/nvme1n1... 2024-03-11T21:54:08.601 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-11T21:54:09.213 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:08 smithi110 ceph-mon[30535]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T21:54:09.213 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:08 smithi110 ceph-mon[30535]: from='osd.6 [v2:172.21.15.110:6816/462580487,v1:172.21.15.110:6817/462580487]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:09.213 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:08 smithi110 ceph-mon[30535]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T21:54:09.213 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:08 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:09.213 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:08 smithi110 ceph-mon[30535]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:09.213 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:08 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:09.213 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:08 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:09.213 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:08 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:09.213 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:08 smithi110 ceph-mon[30535]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:09.384 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[25338]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[25338]: from='osd.6 [v2:172.21.15.110:6816/462580487,v1:172.21.15.110:6817/462580487]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[25338]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[25338]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[25338]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[34177]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T21:54:09.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[34177]: from='osd.6 [v2:172.21.15.110:6816/462580487,v1:172.21.15.110:6817/462580487]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:09.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[34177]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T21:54:09.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:09.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[34177]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:09.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:09.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:09.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:09.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:08 smithi102 ceph-mon[34177]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:09.488 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:54:09 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:54:09.219+0000 7f889ea70700 -1 osd.6 0 waiting for initial osdmap 2024-03-11T21:54:09.488 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:54:09 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:54:09.224+0000 7f889685b700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:54:10.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:09 smithi110 ceph-mon[30535]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:54:10.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:09 smithi110 ceph-mon[30535]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T21:54:10.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:09 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:10.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:09 smithi110 ceph-mon[30535]: from='osd.6 ' entity='osd.6' 2024-03-11T21:54:10.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:09 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:10.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[25338]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:54:10.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[25338]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T21:54:10.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:10.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[25338]: from='osd.6 ' entity='osd.6' 2024-03-11T21:54:10.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[34177]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:54:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[34177]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T21:54:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[34177]: from='osd.6 ' entity='osd.6' 2024-03-11T21:54:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:09 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:11.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:10 smithi110 ceph-mon[30535]: purged_snaps scrub starts 2024-03-11T21:54:11.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:10 smithi110 ceph-mon[30535]: purged_snaps scrub ok 2024-03-11T21:54:11.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:10 smithi110 ceph-mon[30535]: osd.6 [v2:172.21.15.110:6816/462580487,v1:172.21.15.110:6817/462580487] boot 2024-03-11T21:54:11.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:10 smithi110 ceph-mon[30535]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T21:54:11.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:10 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:11.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:10 smithi110 ceph-mon[30535]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[25338]: purged_snaps scrub starts 2024-03-11T21:54:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[25338]: purged_snaps scrub ok 2024-03-11T21:54:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[25338]: osd.6 [v2:172.21.15.110:6816/462580487,v1:172.21.15.110:6817/462580487] boot 2024-03-11T21:54:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[25338]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T21:54:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[25338]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[34177]: purged_snaps scrub starts 2024-03-11T21:54:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[34177]: purged_snaps scrub ok 2024-03-11T21:54:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[34177]: osd.6 [v2:172.21.15.110:6816/462580487,v1:172.21.15.110:6817/462580487] boot 2024-03-11T21:54:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[34177]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T21:54:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:54:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:10 smithi102 ceph-mon[34177]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:54:11.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:11 smithi110 ceph-mon[30535]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T21:54:12.335 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/nvme1n1 2024-03-11T21:54:12.336 INFO:teuthology.orchestra.run.smithi110.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:54:12.336 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-11T21:54:12.336 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-03-11T21:54:12.336 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-03-11T21:54:12.336 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00950215 s, 1.1 GB/s 2024-03-11T21:54:12.336 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-03-11T21:54:12.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:11 smithi102 ceph-mon[25338]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T21:54:12.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:11 smithi102 ceph-mon[34177]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T21:54:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T21:54:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:12.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:12.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:12.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:12.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:12.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:12.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:13.385 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi110:/dev/nvme1n1 2024-03-11T21:54:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T21:54:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:13.620 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:54:14.070 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:13 smithi110 ceph-mon[30535]: Detected new or changed devices on smithi110 2024-03-11T21:54:14.070 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:13 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi110 to 5696M 2024-03-11T21:54:14.070 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:13 smithi110 ceph-mon[30535]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T21:54:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:13 smithi102 ceph-mon[25338]: Detected new or changed devices on smithi110 2024-03-11T21:54:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:13 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi110 to 5696M 2024-03-11T21:54:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:13 smithi102 ceph-mon[25338]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T21:54:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:13 smithi102 ceph-mon[34177]: Detected new or changed devices on smithi110 2024-03-11T21:54:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:13 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi110 to 5696M 2024-03-11T21:54:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:13 smithi102 ceph-mon[34177]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T21:54:15.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:14 smithi110 ceph-mon[30535]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-11T21:54:15.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:14 smithi110 ceph-mon[30535]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T21:54:15.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:14 smithi102 ceph-mon[25338]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-11T21:54:15.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:14 smithi102 ceph-mon[25338]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T21:54:15.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:14 smithi102 ceph-mon[34177]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-11T21:54:15.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:14 smithi102 ceph-mon[34177]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T21:54:16.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:15 smithi110 ceph-mon[30535]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:54:16.298 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:15 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:54:16.298 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:15 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:54:16.298 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:15 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:15 smithi102 ceph-mon[25338]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:54:16.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:15 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:54:16.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:15 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:54:16.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:15 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:15 smithi102 ceph-mon[34177]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:54:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:15 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:54:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:15 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:54:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:15 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:17.357 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:16 smithi110 ceph-mon[30535]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-11T21:54:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:16 smithi102 ceph-mon[25338]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-11T21:54:17.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:16 smithi102 ceph-mon[34177]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-11T21:54:19.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[25338]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/3173532989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800b08ef-0293-491b-a615-0ab0692fd14f"}]: dispatch 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800b08ef-0293-491b-a615-0ab0692fd14f"}]: dispatch 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[25338]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "800b08ef-0293-491b-a615-0ab0692fd14f"}]': finished 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[25338]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[34177]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/3173532989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800b08ef-0293-491b-a615-0ab0692fd14f"}]: dispatch 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800b08ef-0293-491b-a615-0ab0692fd14f"}]: dispatch 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[34177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "800b08ef-0293-491b-a615-0ab0692fd14f"}]': finished 2024-03-11T21:54:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[34177]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T21:54:19.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:19 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:19.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:19 smithi110 ceph-mon[30535]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T21:54:19.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:19 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/3173532989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800b08ef-0293-491b-a615-0ab0692fd14f"}]: dispatch 2024-03-11T21:54:19.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:19 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800b08ef-0293-491b-a615-0ab0692fd14f"}]: dispatch 2024-03-11T21:54:19.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:19 smithi110 ceph-mon[30535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "800b08ef-0293-491b-a615-0ab0692fd14f"}]': finished 2024-03-11T21:54:19.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:19 smithi110 ceph-mon[30535]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T21:54:19.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:19 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:20.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:20 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/2808880011' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:54:20.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:20 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/2808880011' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:54:20.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:20 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/2808880011' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:54:21.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:21 smithi102 ceph-mon[25338]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:21.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:21 smithi102 ceph-mon[34177]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:21.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:21 smithi110 ceph-mon[30535]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:23.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:23 smithi102 ceph-mon[34177]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:23 smithi102 ceph-mon[25338]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:23.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:23 smithi110 ceph-mon[30535]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:25.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:25 smithi102 ceph-mon[34177]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:25 smithi102 ceph-mon[25338]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:25.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:25 smithi110 ceph-mon[30535]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:26 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T21:54:26.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:26 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:26.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:26 smithi102 ceph-mon[25338]: Deploying daemon osd.7 on smithi110 2024-03-11T21:54:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:26 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T21:54:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:26 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:26 smithi102 ceph-mon[34177]: Deploying daemon osd.7 on smithi110 2024-03-11T21:54:26.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:26 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T21:54:26.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:26 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:26.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:26 smithi110 ceph-mon[30535]: Deploying daemon osd.7 on smithi110 2024-03-11T21:54:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:27 smithi102 ceph-mon[25338]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:27.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:27 smithi102 ceph-mon[34177]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:27.688 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:27 smithi110 ceph-mon[30535]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:29.459 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:29 smithi110 ceph-mon[30535]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:29 smithi102 ceph-mon[25338]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:29 smithi102 ceph-mon[34177]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:31.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:31 smithi102 ceph-mon[25338]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:31 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:31 smithi102 ceph-mon[34177]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:31 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:31.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:31 smithi110 ceph-mon[30535]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:31.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:31.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:31.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:31 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:33.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:33 smithi110 ceph-mon[30535]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:33 smithi102 ceph-mon[25338]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:33 smithi102 ceph-mon[34177]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:35.187 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 7 on host 'smithi110' 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='osd.7 [v2:172.21.15.110:6824/2717758054,v1:172.21.15.110:6825/2717758054]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:35 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='osd.7 [v2:172.21.15.110:6824/2717758054,v1:172.21.15.110:6825/2717758054]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='osd.7 [v2:172.21.15.110:6824/2717758054,v1:172.21.15.110:6825/2717758054]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:35 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:35.860 DEBUG:teuthology.orchestra.run.smithi110:osd.7> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T21:54:35.862 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-11T21:54:35.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T21:54:36.107 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:54:36.348 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:36 smithi110 ceph-mon[30535]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T21:54:36.348 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:36 smithi110 ceph-mon[30535]: from='osd.7 [v2:172.21.15.110:6824/2717758054,v1:172.21.15.110:6825/2717758054]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:36.348 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:36 smithi110 ceph-mon[30535]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T21:54:36.348 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:36 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:36.348 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:36 smithi110 ceph-mon[30535]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:36.349 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:54:36 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:54:36.266+0000 7fef0c983700 -1 osd.7 0 waiting for initial osdmap 2024-03-11T21:54:36.349 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:54:36 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:54:36.272+0000 7fef08776700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:54:36.480 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[25338]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T21:54:36.480 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[25338]: from='osd.7 [v2:172.21.15.110:6824/2717758054,v1:172.21.15.110:6825/2717758054]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:36.480 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[25338]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T21:54:36.480 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:36.480 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[25338]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:36.481 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[34177]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T21:54:36.481 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[34177]: from='osd.7 [v2:172.21.15.110:6824/2717758054,v1:172.21.15.110:6825/2717758054]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:36.481 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[34177]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T21:54:36.481 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:36.481 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:36 smithi102 ceph-mon[34177]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-03-11T21:54:37.430 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:37 smithi110 ceph-mon[30535]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:54:37.430 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:37 smithi110 ceph-mon[30535]: osdmap e54: 8 total, 7 up, 8 in 2024-03-11T21:54:37.430 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:37.430 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:37.430 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:37 smithi110 ceph-mon[30535]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:37.430 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:37 smithi110 ceph-mon[30535]: osd.7 [v2:172.21.15.110:6824/2717758054,v1:172.21.15.110:6825/2717758054] boot 2024-03-11T21:54:37.430 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:37 smithi110 ceph-mon[30535]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T21:54:37.430 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:37 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:37.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[25338]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:54:37.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[25338]: osdmap e54: 8 total, 7 up, 8 in 2024-03-11T21:54:37.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:37.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:37.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[25338]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:37.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[25338]: osd.7 [v2:172.21.15.110:6824/2717758054,v1:172.21.15.110:6825/2717758054] boot 2024-03-11T21:54:37.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[25338]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T21:54:37.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:37.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[34177]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-03-11T21:54:37.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[34177]: osdmap e54: 8 total, 7 up, 8 in 2024-03-11T21:54:37.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:37.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:37.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[34177]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:54:37.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[34177]: osd.7 [v2:172.21.15.110:6824/2717758054,v1:172.21.15.110:6825/2717758054] boot 2024-03-11T21:54:37.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[34177]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T21:54:37.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:37 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:54:37.977 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:54:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:38 smithi102 ceph-mon[34177]: purged_snaps scrub starts 2024-03-11T21:54:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:38 smithi102 ceph-mon[34177]: purged_snaps scrub ok 2024-03-11T21:54:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:38 smithi102 ceph-mon[34177]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T21:54:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:38 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/711371969' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:54:38.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:38 smithi102 ceph-mon[25338]: purged_snaps scrub starts 2024-03-11T21:54:38.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:38 smithi102 ceph-mon[25338]: purged_snaps scrub ok 2024-03-11T21:54:38.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:38 smithi102 ceph-mon[25338]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T21:54:38.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:38 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/711371969' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:54:38.704 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1710194076,"num_in_osds":8,"osd_in_since":1710194058,"num_remapped_pgs":1} 2024-03-11T21:54:38.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:54:38.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:38 smithi110 ceph-mon[30535]: purged_snaps scrub starts 2024-03-11T21:54:38.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:38 smithi110 ceph-mon[30535]: purged_snaps scrub ok 2024-03-11T21:54:38.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:38 smithi110 ceph-mon[30535]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T21:54:38.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:38 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/711371969' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:54:38.945 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:54:39.319 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:54:39.319 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T21:54:39.319 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T21:54:39.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:39.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:39.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:39 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:54:39.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T21:54:39.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:54:39.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:39.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:39.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:39.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:39 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:40 smithi102 ceph-mon[34177]: Detected new or changed devices on smithi110 2024-03-11T21:54:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:40 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi110 to 4272M 2024-03-11T21:54:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:40 smithi102 ceph-mon[34177]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T21:54:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:40 smithi102 ceph-mon[25338]: Detected new or changed devices on smithi110 2024-03-11T21:54:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:40 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi110 to 4272M 2024-03-11T21:54:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:40 smithi102 ceph-mon[25338]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T21:54:40.700 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:54:40.701 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":59,"fsid":"0126e6b2-dff1-11ee-95c8-87774f69a715","created":"2024-03-11T21:49:14.830092+0000","modified":"2024-03-11T21:54:40.641291+0000","last_up_change":"2024-03-11T21:54:36.635837+0000","last_in_change":"2024-03-11T21:54:18.389315+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:52:26.615067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1f36f4f8-fe18-4d44-b3df-b07709f4af70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6803","nonce":1026485852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6805","nonce":1026485852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6809","nonce":1026485852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6807","nonce":1026485852}]},"public_addr":"172.21.15.102:6803/1026485852","cluster_addr":"172.21.15.102:6805/1026485852","heartbeat_back_addr":"172.21.15.102:6809/1026485852","heartbeat_front_addr":"172.21.15.102:6807/1026485852","state":["exists","up"]},{"osd":1,"uuid":"9bd35249-7d61-4d04-ba14-31debbf69fd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6811","nonce":265722643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6813","nonce":265722643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6817","nonce":265722643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6815","nonce":265722643}]},"public_addr":"172.21.15.102:6811/265722643","cluster_addr":"172.21.15.102:6813/265722643","heartbeat_back_addr":"172.21.15.102:6817/265722643","heartbeat_front_addr":"172.21.15.102:6815/265722643","state":["exists","up"]},{"osd":2,"uuid":"6c25d672-a2ee-420d-9008-cb48d85c3b57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6819","nonce":2501373355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6821","nonce":2501373355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6825","nonce":2501373355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6823","nonce":2501373355}]},"public_addr":"172.21.15.102:6819/2501373355","cluster_addr":"172.21.15.102:6821/2501373355","heartbeat_back_addr":"172.21.15.102:6825/2501373355","heartbeat_front_addr":"172.21.15.102:6823/2501373355","state":["exists","up"]},{"osd":3,"uuid":"89807137-de78-4a04-8442-478a3e2eb242","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6827","nonce":1356933888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6829","nonce":1356933888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6833","nonce":1356933888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6831","nonce":1356933888}]},"public_addr":"172.21.15.102:6827/1356933888","cluster_addr":"172.21.15.102:6829/1356933888","heartbeat_back_addr":"172.21.15.102:6833/1356933888","heartbeat_front_addr":"172.21.15.102:6831/1356933888","state":["exists","up"]},{"osd":4,"uuid":"c670cc8e-51e7-4b4c-8e20-160b54ab7688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6801","nonce":882287460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6803","nonce":882287460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6807","nonce":882287460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6805","nonce":882287460}]},"public_addr":"172.21.15.110:6801/882287460","cluster_addr":"172.21.15.110:6803/882287460","heartbeat_back_addr":"172.21.15.110:6807/882287460","heartbeat_front_addr":"172.21.15.110:6805/882287460","state":["exists","up"]},{"osd":5,"uuid":"e087753a-eddf-4880-a51e-8efdfec06331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6809","nonce":1539634603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6811","nonce":1539634603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6815","nonce":1539634603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6813","nonce":1539634603}]},"public_addr":"172.21.15.110:6809/1539634603","cluster_addr":"172.21.15.110:6811/1539634603","heartbeat_back_addr":"172.21.15.110:6815/1539634603","heartbeat_front_addr":"172.21.15.110:6813/1539634603","state":["exists","up"]},{"osd":6,"uuid":"e599016e-0eeb-4fdd-a8a9-6fc8eed1218a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6817","nonce":462580487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6819","nonce":462580487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6823","nonce":462580487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6821","nonce":462580487}]},"public_addr":"172.21.15.110:6817/462580487","cluster_addr":"172.21.15.110:6819/462580487","heartbeat_back_addr":"172.21.15.110:6823/462580487","heartbeat_front_addr":"172.21.15.110:6821/462580487","state":["exists","up"]},{"osd":7,"uuid":"800b08ef-0293-491b-a615-0ab0692fd14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6825","nonce":2717758054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6827","nonce":2717758054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6831","nonce":2717758054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6829","nonce":2717758054}]},"public_addr":"172.21.15.110:6825/2717758054","cluster_addr":"172.21.15.110:6827/2717758054","heartbeat_back_addr":"172.21.15.110:6831/2717758054","heartbeat_front_addr":"172.21.15.110:6829/2717758054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:51:31.165874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:51:57.295234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:52:23.000112+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:52:49.157367+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:53:13.758729+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:53:40.969407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:54:08.014278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:54:35.396853+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2821019614":"2024-03-12T21:50:12.284168+0000","172.21.15.102:0/3686793380":"2024-03-12T21:50:12.284168+0000","172.21.15.102:0/432123271":"2024-03-12T21:50:12.284168+0000","172.21.15.102:6801/2761823548":"2024-03-12T21:50:12.284168+0000","172.21.15.102:6801/161836558":"2024-03-12T21:49:47.015350+0000","172.21.15.102:0/3333959561":"2024-03-12T21:49:47.015350+0000","172.21.15.102:6800/2761823548":"2024-03-12T21:50:12.284168+0000","172.21.15.102:6800/161836558":"2024-03-12T21:49:47.015350+0000","172.21.15.102:0/4278625363":"2024-03-12T21:49:47.015350+0000","172.21.15.102:0/3679135738":"2024-03-12T21:49:47.015350+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:54:40.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:40 smithi110 ceph-mon[30535]: Detected new or changed devices on smithi110 2024-03-11T21:54:40.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:40 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi110 to 4272M 2024-03-11T21:54:40.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:40 smithi110 ceph-mon[30535]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T21:54:41.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:41 smithi102 ceph-mon[25338]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T21:54:41.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:41 smithi102 ceph-mon[25338]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T21:54:41.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:41 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/521206849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:54:41.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:41 smithi102 ceph-mon[34177]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T21:54:41.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:41 smithi102 ceph-mon[34177]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T21:54:41.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:41 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/521206849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:54:41.430 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:52:26.615067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T21:54:41.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T21:54:41.672 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:54:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:41 smithi110 ceph-mon[30535]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T21:54:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:41 smithi110 ceph-mon[30535]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T21:54:41.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:41 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/521206849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:54:43.410 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-11T21:54:43.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:43 smithi102 ceph-mon[25338]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-11T21:54:43.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:43 smithi102 ceph-mon[34177]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-11T21:54:43.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:43 smithi110 ceph-mon[30535]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-11T21:54:43.985 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi102 2024-03-11T21:54:43.986 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi102=foo.a' 2024-03-11T21:54:44.236 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:54:44.613 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:44 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/865564217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:54:44.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:44 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/865564217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:54:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:44 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/865564217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:54:45.552 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:45 smithi110 ceph-mon[30535]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:54:45.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:45 smithi102 ceph-mon[25338]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:54:45.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:45 smithi102 ceph-mon[34177]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:54:45.960 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled rgw.foo.a update... 2024-03-11T21:54:46.518 DEBUG:teuthology.orchestra.run.smithi102:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T21:54:46.520 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi110 2024-03-11T21:54:46.520 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-11T21:54:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:54:47.143 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:54:47.143 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-11T21:54:47.143 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:47.143 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:47.143 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:47.143 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:47.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:47.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:54:47.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:54:47.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:47.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:47.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: Deploying daemon rgw.foo.a on smithi102 2024-03-11T21:54:47.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:46 smithi110 ceph-mon[30535]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-11T21:54:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:54:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-11T21:54:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: Deploying daemon rgw.foo.a on smithi102 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[34177]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:47.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:47.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:54:47.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:54:47.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:47.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:47.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: Deploying daemon rgw.foo.a on smithi102 2024-03-11T21:54:47.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:46 smithi102 ceph-mon[25338]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-11T21:54:48.403 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:54:48 smithi102 systemd[1]: Starting Ceph rgw.foo.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:54:48.664 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:54:48 smithi102 podman[70207]: 2024-03-11 21:54:48.409825445 +0000 UTC m=+0.111075548 container create 7543b2522bb338bc367d60cc126d2fe5fb4089917d34e698f4d930df952eaebf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-11T21:54:48.664 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:54:48 smithi102 podman[70207]: 2024-03-11 21:54:48.316664887 +0000 UTC m=+0.017915001 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:54:48.664 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:54:48 smithi102 podman[70207]: 2024-03-11 21:54:48.670802841 +0000 UTC m=+0.372052959 container init 7543b2522bb338bc367d60cc126d2fe5fb4089917d34e698f4d930df952eaebf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:54:48.919 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:54:48 smithi102 podman[70207]: 2024-03-11 21:54:48.673942435 +0000 UTC m=+0.375192547 container start 7543b2522bb338bc367d60cc126d2fe5fb4089917d34e698f4d930df952eaebf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:54:48.919 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:54:48 smithi102 bash[70207]: 7543b2522bb338bc367d60cc126d2fe5fb4089917d34e698f4d930df952eaebf 2024-03-11T21:54:48.919 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:54:48 smithi102 systemd[1]: Started Ceph rgw.foo.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:54:49.393 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'datapool' created 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[34177]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/2302079610' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[34177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:49.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[25338]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:54:49.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/2302079610' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:54:49.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[25338]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:54:49.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:49 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:49.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:49 smithi110 ceph-mon[30535]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:54:49.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:49 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/2302079610' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:54:49.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:49 smithi110 ceph-mon[30535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:54:49.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:49.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:49 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:54:50.102 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- rbd pool init datapool 2024-03-11T21:54:50.339 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:54:50.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[25338]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-11T21:54:50.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[25338]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T21:54:50.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[25338]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T21:54:50.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/681732024' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:54:50.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:54:50.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[34177]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-11T21:54:50.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[34177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T21:54:50.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[34177]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T21:54:50.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/681732024' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:54:50.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:50 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:54:50.690 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:50 smithi110 ceph-mon[30535]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-11T21:54:50.690 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:50 smithi110 ceph-mon[30535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T21:54:50.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:50 smithi110 ceph-mon[30535]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T21:54:50.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:50 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/681732024' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:54:50.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:50 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:54:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[25338]: pgmap v189: 36 pgs: 3 creating+peering, 1 active+clean, 32 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:54:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[25338]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:54:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:54:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[25338]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T21:54:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:51.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:51.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[34177]: pgmap v189: 36 pgs: 3 creating+peering, 1 active+clean, 32 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:54:51.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[34177]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:54:51.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:54:51.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[34177]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T21:54:51.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:51.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:51.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:51.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:51 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:51.718 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:51 smithi110 ceph-mon[30535]: pgmap v189: 36 pgs: 3 creating+peering, 1 active+clean, 32 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:54:51.718 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:51 smithi110 ceph-mon[30535]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:54:51.718 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:51 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:54:51.718 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:51 smithi110 ceph-mon[30535]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T21:54:51.718 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:51 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:51.718 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:51 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:51.718 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:51 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:51.719 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:51 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:54:52.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[25338]: Checking dashboard <-> RGW credentials 2024-03-11T21:54:52.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[25338]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T21:54:52.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[25338]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/784574092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:54:52.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[25338]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:54:52.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:52.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[34177]: Checking dashboard <-> RGW credentials 2024-03-11T21:54:52.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[34177]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T21:54:52.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[34177]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/784574092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:54:52.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[34177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:54:52.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:52 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:52.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:52 smithi110 ceph-mon[30535]: Checking dashboard <-> RGW credentials 2024-03-11T21:54:52.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:52 smithi110 ceph-mon[30535]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T21:54:52.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:52 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:52 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:52 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:52 smithi110 ceph-mon[30535]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:54:52.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:52 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/784574092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:54:52.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:52 smithi110 ceph-mon[30535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:54:52.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:52 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[25338]: pgmap v192: 68 pgs: 19 creating+peering, 21 active+clean, 28 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-11T21:54:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:54:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[25338]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:54:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[25338]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T21:54:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[25338]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T21:54:53.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[34177]: pgmap v192: 68 pgs: 19 creating+peering, 21 active+clean, 28 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-11T21:54:53.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:54:53.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[34177]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:54:53.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[34177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T21:54:53.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:53 smithi102 ceph-mon[34177]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T21:54:53.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:53 smithi110 ceph-mon[30535]: pgmap v192: 68 pgs: 19 creating+peering, 21 active+clean, 28 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-11T21:54:53.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:53 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:54:53.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:53 smithi110 ceph-mon[30535]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:54:53.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:53 smithi110 ceph-mon[30535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T21:54:53.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:53 smithi110 ceph-mon[30535]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T21:54:54.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:54 smithi110 ceph-mon[30535]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T21:54:54.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:54 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:54 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:54 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:54 smithi110 ceph-mon[30535]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[25338]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T21:54:54.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[25338]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[34177]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T21:54:54.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:54.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:54 smithi102 ceph-mon[34177]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:54:55.101 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.110 --placement '1;smithi110=iscsi.a' 2024-03-11T21:54:55.344 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:54:55.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:55 smithi110 ceph-mon[30535]: pgmap v195: 100 pgs: 22 creating+peering, 40 active+clean, 38 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-11T21:54:55.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:55 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:54:55.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:55 smithi110 ceph-mon[30535]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:54:55.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:55 smithi110 ceph-mon[30535]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T21:54:55.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:55 smithi102 ceph-mon[25338]: pgmap v195: 100 pgs: 22 creating+peering, 40 active+clean, 38 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-11T21:54:55.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:55 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:54:55.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:55 smithi102 ceph-mon[25338]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:54:55.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:55 smithi102 ceph-mon[25338]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T21:54:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:55 smithi102 ceph-mon[34177]: pgmap v195: 100 pgs: 22 creating+peering, 40 active+clean, 38 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-11T21:54:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:55 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:54:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:55 smithi102 ceph-mon[34177]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:54:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:55 smithi102 ceph-mon[34177]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T21:54:56.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:56 smithi110 ceph-mon[30535]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T21:54:56.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:56 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:56 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:56 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:56 smithi110 ceph-mon[30535]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[25338]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T21:54:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[25338]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[34177]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T21:54:56.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:56.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:56 smithi102 ceph-mon[34177]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:54:57.166 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled iscsi.datapool update... 2024-03-11T21:54:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: pgmap v198: 132 pgs: 19 creating+peering, 64 active+clean, 49 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T21:54:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:54:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:54:57.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T21:54:57.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:54:57.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:57 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:57.849 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-11T21:54:57.849 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:54:57.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T21:54:57.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: pgmap v198: 132 pgs: 19 creating+peering, 64 active+clean, 49 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T21:54:57.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: pgmap v198: 132 pgs: 19 creating+peering, 64 active+clean, 49 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:54:57.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:54:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T21:54:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/280541144' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1911152998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:54:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:54:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:57 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:57.880 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:54:57.880 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T21:54:57.910 DEBUG:teuthology.orchestra.run.smithi110:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@iscsi.iscsi.a.service 2024-03-11T21:54:57.952 INFO:tasks.cephadm:Adding prometheus.a on smithi110 2024-03-11T21:54:57.953 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi110=a' 2024-03-11T21:54:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:54:58.606 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.110", "placement": "1;smithi110=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: Saving service iscsi.datapool spec with placement smithi110=iscsi.a;count:1 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T21:54:58.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:58 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.110", "placement": "1;smithi110=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:54:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: Saving service iscsi.datapool spec with placement smithi110=iscsi.a;count:1 2024-03-11T21:54:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:54:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:54:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T21:54:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.110", "placement": "1;smithi110=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: Saving service iscsi.datapool spec with placement smithi110=iscsi.a;count:1 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T21:54:58.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T21:54:58.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:58 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:54:59.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:59 smithi110 ceph-mon[30535]: Deploying daemon iscsi.iscsi.a on smithi110 2024-03-11T21:54:59.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:59 smithi110 ceph-mon[30535]: pgmap v201: 132 pgs: 1 creating+activating, 3 creating+peering, 118 active+clean, 10 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 18 op/s 2024-03-11T21:54:59.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:59 smithi110 ceph-mon[30535]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:54:59.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:54:59 smithi110 ceph-mon[30535]: Cluster is now healthy 2024-03-11T21:54:59.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:59 smithi102 ceph-mon[25338]: Deploying daemon iscsi.iscsi.a on smithi110 2024-03-11T21:54:59.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:59 smithi102 ceph-mon[25338]: pgmap v201: 132 pgs: 1 creating+activating, 3 creating+peering, 118 active+clean, 10 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 18 op/s 2024-03-11T21:54:59.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:59 smithi102 ceph-mon[25338]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:54:59.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:54:59 smithi102 ceph-mon[25338]: Cluster is now healthy 2024-03-11T21:54:59.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:59 smithi102 ceph-mon[34177]: Deploying daemon iscsi.iscsi.a on smithi110 2024-03-11T21:54:59.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:59 smithi102 ceph-mon[34177]: pgmap v201: 132 pgs: 1 creating+activating, 3 creating+peering, 118 active+clean, 10 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 18 op/s 2024-03-11T21:54:59.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:59 smithi102 ceph-mon[34177]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:54:59.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:54:59 smithi102 ceph-mon[34177]: Cluster is now healthy 2024-03-11T21:55:00.606 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled prometheus update... 2024-03-11T21:55:01.705 DEBUG:teuthology.orchestra.run.smithi110:prometheus.a> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T21:55:01.707 INFO:tasks.cephadm:Adding node-exporter.a on smithi102 2024-03-11T21:55:01.707 INFO:tasks.cephadm:Adding node-exporter.b on smithi110 2024-03-11T21:55:01.707 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi102=a;smithi110=b' 2024-03-11T21:55:01.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:01 smithi110 ceph-mon[30535]: pgmap v202: 132 pgs: 1 creating+activating, 129 active+clean, 2 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.4 KiB/s wr, 76 op/s 2024-03-11T21:55:01.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:01 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:01.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:01 smithi102 ceph-mon[25338]: pgmap v202: 132 pgs: 1 creating+activating, 129 active+clean, 2 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.4 KiB/s wr, 76 op/s 2024-03-11T21:55:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:01 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:01.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:01 smithi102 ceph-mon[34177]: pgmap v202: 132 pgs: 1 creating+activating, 129 active+clean, 2 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.4 KiB/s wr, 76 op/s 2024-03-11T21:55:01.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:01 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:01.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:01 smithi110 podman[66588]: 2024-03-11 21:55:01.889249082 +0000 UTC m=+0.132298622 container create 6e2c30803a349f234eb5d82936be9c3446b28d177c60dd0588e5894997a6a16d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-11T21:55:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:55:02.360 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 podman[66588]: 2024-03-11 21:55:02.116554708 +0000 UTC m=+0.359604259 container init 6e2c30803a349f234eb5d82936be9c3446b28d177c60dd0588e5894997a6a16d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:55:02.360 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 podman[66588]: 2024-03-11 21:55:02.120008231 +0000 UTC m=+0.363057779 container start 6e2c30803a349f234eb5d82936be9c3446b28d177c60dd0588e5894997a6a16d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T21:55:02.360 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 bash[66588]: 6e2c30803a349f234eb5d82936be9c3446b28d177c60dd0588e5894997a6a16d 2024-03-11T21:55:02.360 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 systemd[1]: Started Ceph iscsi.iscsi.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:55:02.610 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug Started the configuration object watcher 2024-03-11T21:55:02.610 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug Checking for config object changes every 1s 2024-03-11T21:55:02.610 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug Processing osd blocklist entries for this node 2024-03-11T21:55:02.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-mon[30535]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi110=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:02.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-mon[30535]: Saving service prometheus spec with placement smithi110=a;count:1 2024-03-11T21:55:02.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-mon[30535]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T21:55:02.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[25338]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi110=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[25338]: Saving service prometheus spec with placement smithi110=a;count:1 2024-03-11T21:55:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[25338]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T21:55:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:02.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[34177]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi110=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:02.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[34177]: Saving service prometheus spec with placement smithi110=a;count:1 2024-03-11T21:55:02.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[34177]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T21:55:02.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:02.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:02 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:02.924 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug Reading the configuration object to update local LIO configuration 2024-03-11T21:55:02.924 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug Configuration does not have an entry for this host(smithi110.front.sepia.ceph.com) - nothing to define to LIO 2024-03-11T21:55:02.924 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:02 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-11T21:55:03.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:03 smithi110 ceph-mon[30535]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T21:55:03.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:03 smithi110 ceph-mon[30535]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 6.0 KiB/s wr, 184 op/s 2024-03-11T21:55:03.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:03 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/3437530654' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T21:55:03.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:03 smithi102 ceph-mon[25338]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T21:55:03.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:03 smithi102 ceph-mon[25338]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 6.0 KiB/s wr, 184 op/s 2024-03-11T21:55:03.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:03 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/3437530654' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T21:55:03.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:03 smithi102 ceph-mon[34177]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T21:55:03.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:03 smithi102 ceph-mon[34177]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 6.0 KiB/s wr, 184 op/s 2024-03-11T21:55:03.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:03 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/3437530654' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T21:55:04.517 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled node-exporter update... 2024-03-11T21:55:05.681 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:05 smithi110 ceph-mon[30535]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.6 KiB/s wr, 141 op/s 2024-03-11T21:55:05.682 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:05 smithi110 ceph-mon[30535]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T21:55:05.682 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:05 smithi110 ceph-mon[30535]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi110=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:05.682 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:05 smithi110 ceph-mon[30535]: Saving service node-exporter spec with placement smithi102=a;smithi110=b;count:2 2024-03-11T21:55:05.682 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:05 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:05.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[25338]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.6 KiB/s wr, 141 op/s 2024-03-11T21:55:05.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[25338]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T21:55:05.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[25338]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi110=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:05.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[25338]: Saving service node-exporter spec with placement smithi102=a;smithi110=b;count:2 2024-03-11T21:55:05.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:05.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[34177]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.6 KiB/s wr, 141 op/s 2024-03-11T21:55:05.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[34177]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T21:55:05.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[34177]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi110=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:05.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[34177]: Saving service node-exporter spec with placement smithi102=a;smithi110=b;count:2 2024-03-11T21:55:05.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:05 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:05.975 DEBUG:teuthology.orchestra.run.smithi102:node-exporter.a> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T21:55:05.978 DEBUG:teuthology.orchestra.run.smithi110:node-exporter.b> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T21:55:05.980 INFO:tasks.cephadm:Adding alertmanager.a on smithi102 2024-03-11T21:55:05.981 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi102=a' 2024-03-11T21:55:06.902 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:55:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:07 smithi110 ceph-mon[30535]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.0 KiB/s wr, 125 op/s 2024-03-11T21:55:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:55:07.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:55:07.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:07 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[34177]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.0 KiB/s wr, 125 op/s 2024-03-11T21:55:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:55:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:55:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:07.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[25338]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.0 KiB/s wr, 125 op/s 2024-03-11T21:55:07.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:07.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:07.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:55:07.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:55:07.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:07 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:08.655 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled alertmanager update... 2024-03-11T21:55:08.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:08 smithi110 ceph-mon[30535]: Deploying daemon prometheus.a on smithi110 2024-03-11T21:55:08.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:08 smithi102 ceph-mon[34177]: Deploying daemon prometheus.a on smithi110 2024-03-11T21:55:08.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:08 smithi102 ceph-mon[25338]: Deploying daemon prometheus.a on smithi110 2024-03-11T21:55:09.464 DEBUG:teuthology.orchestra.run.smithi102:alertmanager.a> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T21:55:09.467 INFO:tasks.cephadm:Adding grafana.a on smithi110 2024-03-11T21:55:09.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi110=a' 2024-03-11T21:55:09.716 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:55:09.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:09 smithi110 ceph-mon[30535]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.3 KiB/s wr, 106 op/s 2024-03-11T21:55:09.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:09 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:09.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:09 smithi102 ceph-mon[25338]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.3 KiB/s wr, 106 op/s 2024-03-11T21:55:09.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:09 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:09.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:09 smithi102 ceph-mon[34177]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.3 KiB/s wr, 106 op/s 2024-03-11T21:55:09.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:09 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:10.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:10 smithi110 ceph-mon[30535]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:10.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:10 smithi110 ceph-mon[30535]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-03-11T21:55:10.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:10 smithi102 ceph-mon[25338]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:10.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:10 smithi102 ceph-mon[25338]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-03-11T21:55:10.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:10 smithi102 ceph-mon[34177]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:10.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:10 smithi102 ceph-mon[34177]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-03-11T21:55:11.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:11 smithi110 ceph-mon[30535]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.0 KiB/s wr, 69 op/s 2024-03-11T21:55:11.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:11 smithi102 ceph-mon[25338]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.0 KiB/s wr, 69 op/s 2024-03-11T21:55:11.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:11 smithi102 ceph-mon[34177]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.0 KiB/s wr, 69 op/s 2024-03-11T21:55:12.438 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled grafana update... 2024-03-11T21:55:12.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:12 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:55:12.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-11T21:55:12.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-11T21:55:12.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-11T21:55:12.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:12 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-11T21:55:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-11T21:55:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-11T21:55:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:12 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:12.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-11T21:55:12.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-11T21:55:12.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-11T21:55:12.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:12 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:13.379 DEBUG:teuthology.orchestra.run.smithi110:grafana.a> sudo journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T21:55:13.382 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T21:55:13.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:55:13.633 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:13.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[34177]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 861 B/s wr, 23 op/s 2024-03-11T21:55:13.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[34177]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:13.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[34177]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi110=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:13.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[34177]: Saving service grafana spec with placement smithi110=a;count:1 2024-03-11T21:55:13.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-11T21:55:13.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-11T21:55:13.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-11T21:55:13.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[34177]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T21:55:13.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[25338]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 861 B/s wr, 23 op/s 2024-03-11T21:55:13.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[25338]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:13.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[25338]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi110=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:13.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[25338]: Saving service grafana spec with placement smithi110=a;count:1 2024-03-11T21:55:13.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-11T21:55:13.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-11T21:55:13.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-11T21:55:13.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:13 smithi102 ceph-mon[25338]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T21:55:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:13 smithi110 ceph-mon[30535]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 861 B/s wr, 23 op/s 2024-03-11T21:55:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:13 smithi110 ceph-mon[30535]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:13.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:13 smithi110 ceph-mon[30535]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi110=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:13.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:13 smithi110 ceph-mon[30535]: Saving service grafana spec with placement smithi110=a;count:1 2024-03-11T21:55:13.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-11T21:55:13.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-11T21:55:13.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:13 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-11T21:55:13.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:13 smithi110 ceph-mon[30535]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T21:55:14.813 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:14 smithi102 ceph-mon[25338]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T21:55:14.814 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:14 smithi102 ceph-mon[34177]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T21:55:14.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:14 smithi110 ceph-mon[30535]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T21:55:15.621 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-03-11T21:55:15.621 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDDfe9lmC/8JBAADpoLtMzbYG+AthZVIBx1Lg== 2024-03-11T21:55:15.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:15 smithi102 ceph-mon[25338]: pgmap v212: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-11T21:55:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:15 smithi102 ceph-mon[34177]: pgmap v212: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-11T21:55:15.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:15 smithi110 ceph-mon[30535]: pgmap v212: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-11T21:55:16.313 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:55:16.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T21:55:16.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T21:55:16.361 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:55:16.556 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:16 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/3666482598' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:16.556 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:16 smithi102 ceph-mon[25338]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:16.556 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:16 smithi102 ceph-mon[25338]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:55:16.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:16 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/3666482598' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:16.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:16 smithi102 ceph-mon[34177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:16.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:16 smithi102 ceph-mon[34177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:55:16.654 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:55:16.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:16 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/3666482598' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:16.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:16 smithi110 ceph-mon[30535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:16.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:16 smithi110 ceph-mon[30535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:55:17.644 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:17 smithi110 ceph-mon[30535]: pgmap v213: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 41 B/s, 0 objects/s recovering 2024-03-11T21:55:17.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:17 smithi102 ceph-mon[25338]: pgmap v213: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 41 B/s, 0 objects/s recovering 2024-03-11T21:55:17.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:17 smithi102 ceph-mon[34177]: pgmap v213: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 41 B/s, 0 objects/s recovering 2024-03-11T21:55:19.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:19 smithi110 ceph-mon[30535]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 99 B/s, 2 objects/s recovering 2024-03-11T21:55:19.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:19 smithi102 ceph-mon[25338]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 99 B/s, 2 objects/s recovering 2024-03-11T21:55:19.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:19 smithi102 ceph-mon[34177]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 99 B/s, 2 objects/s recovering 2024-03-11T21:55:21.389 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:21 smithi110 systemd[1]: Starting Ceph prometheus.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:55:21.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:21 smithi110 ceph-mon[30535]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 99 B/s, 2 objects/s recovering 2024-03-11T21:55:21.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:21 smithi110 ceph-mon[30535]: from='client.? 172.21.15.110:0/1542829321' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:21.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:21 smithi110 ceph-mon[30535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:21.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:21 smithi110 ceph-mon[30535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:55:21.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:21 smithi102 ceph-mon[25338]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 99 B/s, 2 objects/s recovering 2024-03-11T21:55:21.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:21 smithi102 ceph-mon[25338]: from='client.? 172.21.15.110:0/1542829321' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:21.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:21 smithi102 ceph-mon[25338]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:21.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:21 smithi102 ceph-mon[25338]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:55:21.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:21 smithi102 ceph-mon[34177]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 99 B/s, 2 objects/s recovering 2024-03-11T21:55:21.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:21 smithi102 ceph-mon[34177]: from='client.? 172.21.15.110:0/1542829321' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:21.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:21 smithi102 ceph-mon[34177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:55:21.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:21 smithi102 ceph-mon[34177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:55:21.994 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2024-03-11T21:55:21.994 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDJfe9lJAlZFxAAXT0kJvUuQjVKwe5vkHIQ7g== 2024-03-11T21:55:22.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:22 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:55:23.228 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-11T21:55:23.229 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T21:55:23.229 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T21:55:23.276 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T21:55:23.277 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T21:55:23.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T21:55:23.488 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 podman[68190]: 2024-03-11 21:55:23.308582467 +0000 UTC m=+0.020124774 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-11T21:55:23.488 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 podman[68190]: 2024-03-11 21:55:23.41462718 +0000 UTC m=+0.126169475 container create ae0fd8d4739ea7a7aea61009c0ee713bc70fff17b35626cb9cfa9ffa80332a2f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:55:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:23.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:23 smithi102 ceph-mon[25338]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 0 op/s; 81 B/s, 1 objects/s recovering 2024-03-11T21:55:23.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:23 smithi102 ceph-mon[25338]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:23.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:23 smithi102 ceph-mon[34177]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 0 op/s; 81 B/s, 1 objects/s recovering 2024-03-11T21:55:23.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:23 smithi102 ceph-mon[34177]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:23.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-mon[30535]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 0 op/s; 81 B/s, 1 objects/s recovering 2024-03-11T21:55:23.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-mon[30535]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:23.989 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 podman[68190]: 2024-03-11 21:55:23.625066716 +0000 UTC m=+0.336609010 container init ae0fd8d4739ea7a7aea61009c0ee713bc70fff17b35626cb9cfa9ffa80332a2f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:55:23.989 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 podman[68190]: 2024-03-11 21:55:23.627574674 +0000 UTC m=+0.339116965 container start ae0fd8d4739ea7a7aea61009c0ee713bc70fff17b35626cb9cfa9ffa80332a2f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:55:23.989 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.651Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T21:55:23.989 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.651Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T21:55:23.989 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.651Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi110 (none))" 2024-03-11T21:55:23.989 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.651Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T21:55:23.989 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.651Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.652Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.652Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.654Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.654Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.656Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.656Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.572µs 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.656Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.656Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.656Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.37µs wal_replay_duration=273.032µs wbl_replay_duration=118ns total_replay_duration=326.661µs 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.657Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.657Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.657Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T21:55:23.990 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.673Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.408121ms db_storage=790ns remote_storage=1.16µs web_handler=340ns query_engine=539ns scrape=182.08µs scrape_sd=64.94µs notify=718ns notify_sd=1.251µs rules=15.986815ms tracing=6.777µs 2024-03-11T21:55:23.991 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.673Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T21:55:23.991 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:55:23.673Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T21:55:23.991 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 bash[68190]: ae0fd8d4739ea7a7aea61009c0ee713bc70fff17b35626cb9cfa9ffa80332a2f 2024-03-11T21:55:23.991 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:55:23 smithi110 systemd[1]: Started Ceph prometheus.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:55:24.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ignoring --setuser ceph since I am not root 2024-03-11T21:55:24.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ignoring --setgroup ceph since I am not root 2024-03-11T21:55:24.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:24.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:24.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:24.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:55:24.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[25338]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s; 73 B/s, 1 objects/s recovering 2024-03-11T21:55:24.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:24.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:24.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:24.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:55:24.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:24 smithi102 ceph-mon[34177]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s; 73 B/s, 1 objects/s recovering 2024-03-11T21:55:25.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:24 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:25.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:24 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:25.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:24 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' 2024-03-11T21:55:25.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:24 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:24 smithi110 ceph-mon[30535]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s; 73 B/s, 1 objects/s recovering 2024-03-11T21:55:25.239 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:24 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: ignoring --setuser ceph since I am not root 2024-03-11T21:55:25.239 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:24 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: ignoring --setgroup ceph since I am not root 2024-03-11T21:55:25.239 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:24 smithi110 ceph-mgr[32417]: -- 172.21.15.110:0/2947437398 <== mon.2 v2:172.21.15.102:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5639965341a0 con 0x56399650a800 2024-03-11T21:55:25.522 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:55:25.523 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:25 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:25.371+0000 7f7eeeef0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:55:25.523 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:25 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:25.468+0000 7f7eeeef0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:55:25.738 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:25 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:25.341+0000 7ff2f1a79200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:55:25.738 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:25 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:25.433+0000 7ff2f1a79200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:55:26.052 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:25 smithi110 ceph-mon[30535]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:55:26.052 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:25 smithi110 ceph-mon[30535]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T21:55:26.052 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:25 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/2762818189' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:55:26.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:25 smithi102 ceph-mon[25338]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:55:26.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:25 smithi102 ceph-mon[25338]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T21:55:26.099 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:25 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2762818189' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:55:26.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:25 smithi102 ceph-mon[34177]: from='mgr.14150 172.21.15.102:0/2590948276' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:55:26.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:25 smithi102 ceph-mon[34177]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T21:55:26.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:25 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/2762818189' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:55:26.381 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:26.058+0000 7ff2f1a79200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:55:26.381 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:26.152+0000 7ff2f1a79200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:55:26.381 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:26.293+0000 7ff2f1a79200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:55:26.382 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:26.388+0000 7ff2f1a79200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:55:26.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:26 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:26.116+0000 7f7eeeef0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:55:26.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:26 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:26.210+0000 7f7eeeef0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:55:26.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:26 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:26.353+0000 7f7eeeef0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:55:26.510 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":630575485},{"type":"v1","addr":"172.21.15.102:6801","nonce":630575485}]},"active_addr":"172.21.15.102:6801/630575485","active_change":"2024-03-11T21:50:12.284273+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.102:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2936311452}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1618956400}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3129234549}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":557510979}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":303278813}]}]} 2024-03-11T21:55:26.516 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T21:55:26.516 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T21:55:26.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:55:26.725 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:26.471+0000 7ff2f1a79200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:55:26.763 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:26.813 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:26 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:26.451+0000 7f7eeeef0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:55:26.813 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:26 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:26.539+0000 7f7eeeef0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:55:26.813 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:26 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:26.819+0000 7f7eeeef0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:55:26.988 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:26.732+0000 7ff2f1a79200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:55:26.988 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:26.841+0000 7ff2f1a79200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:55:26.988 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:26.929+0000 7ff2f1a79200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:55:27.115 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:26 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:26.931+0000 7f7eeeef0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:55:27.115 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:27 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:27.023+0000 7f7eeeef0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:55:27.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:27 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:27.121+0000 7f7eeeef0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:55:27.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:27 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:27.304+0000 7f7eeeef0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:55:27.488 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:27 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:27.023+0000 7ff2f1a79200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:55:27.488 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:27 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:27.199+0000 7ff2f1a79200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:55:28.488 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:28 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:28.152+0000 7ff2f1a79200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:55:28.488 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:28 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:28.283+0000 7ff2f1a79200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:55:28.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:28 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:28.285+0000 7f7eeeef0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:55:28.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:28 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:28.419+0000 7f7eeeef0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:55:28.727 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:55:28.727 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":71,"fsid":"0126e6b2-dff1-11ee-95c8-87774f69a715","created":"2024-03-11T21:49:14.830092+0000","modified":"2024-03-11T21:55:13.495656+0000","last_up_change":"2024-03-11T21:54:36.635837+0000","last_in_change":"2024-03-11T21:54:18.389315+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:52:26.615067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-11T21:54:48.886488+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-11T21:54:49.119968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:54:50.734942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:54:52.673627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:54:54.747224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1f36f4f8-fe18-4d44-b3df-b07709f4af70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6803","nonce":1026485852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6805","nonce":1026485852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6809","nonce":1026485852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6807","nonce":1026485852}]},"public_addr":"172.21.15.102:6803/1026485852","cluster_addr":"172.21.15.102:6805/1026485852","heartbeat_back_addr":"172.21.15.102:6809/1026485852","heartbeat_front_addr":"172.21.15.102:6807/1026485852","state":["exists","up"]},{"osd":1,"uuid":"9bd35249-7d61-4d04-ba14-31debbf69fd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6811","nonce":265722643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6813","nonce":265722643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6817","nonce":265722643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6815","nonce":265722643}]},"public_addr":"172.21.15.102:6811/265722643","cluster_addr":"172.21.15.102:6813/265722643","heartbeat_back_addr":"172.21.15.102:6817/265722643","heartbeat_front_addr":"172.21.15.102:6815/265722643","state":["exists","up"]},{"osd":2,"uuid":"6c25d672-a2ee-420d-9008-cb48d85c3b57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6819","nonce":2501373355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6821","nonce":2501373355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6825","nonce":2501373355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6823","nonce":2501373355}]},"public_addr":"172.21.15.102:6819/2501373355","cluster_addr":"172.21.15.102:6821/2501373355","heartbeat_back_addr":"172.21.15.102:6825/2501373355","heartbeat_front_addr":"172.21.15.102:6823/2501373355","state":["exists","up"]},{"osd":3,"uuid":"89807137-de78-4a04-8442-478a3e2eb242","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6827","nonce":1356933888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6829","nonce":1356933888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6833","nonce":1356933888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6831","nonce":1356933888}]},"public_addr":"172.21.15.102:6827/1356933888","cluster_addr":"172.21.15.102:6829/1356933888","heartbeat_back_addr":"172.21.15.102:6833/1356933888","heartbeat_front_addr":"172.21.15.102:6831/1356933888","state":["exists","up"]},{"osd":4,"uuid":"c670cc8e-51e7-4b4c-8e20-160b54ab7688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6801","nonce":882287460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6803","nonce":882287460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6807","nonce":882287460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6805","nonce":882287460}]},"public_addr":"172.21.15.110:6801/882287460","cluster_addr":"172.21.15.110:6803/882287460","heartbeat_back_addr":"172.21.15.110:6807/882287460","heartbeat_front_addr":"172.21.15.110:6805/882287460","state":["exists","up"]},{"osd":5,"uuid":"e087753a-eddf-4880-a51e-8efdfec06331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6809","nonce":1539634603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6811","nonce":1539634603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6815","nonce":1539634603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6813","nonce":1539634603}]},"public_addr":"172.21.15.110:6809/1539634603","cluster_addr":"172.21.15.110:6811/1539634603","heartbeat_back_addr":"172.21.15.110:6815/1539634603","heartbeat_front_addr":"172.21.15.110:6813/1539634603","state":["exists","up"]},{"osd":6,"uuid":"e599016e-0eeb-4fdd-a8a9-6fc8eed1218a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6817","nonce":462580487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6819","nonce":462580487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6823","nonce":462580487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6821","nonce":462580487}]},"public_addr":"172.21.15.110:6817/462580487","cluster_addr":"172.21.15.110:6819/462580487","heartbeat_back_addr":"172.21.15.110:6823/462580487","heartbeat_front_addr":"172.21.15.110:6821/462580487","state":["exists","up"]},{"osd":7,"uuid":"800b08ef-0293-491b-a615-0ab0692fd14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6825","nonce":2717758054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6827","nonce":2717758054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6831","nonce":2717758054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6829","nonce":2717758054}]},"public_addr":"172.21.15.110:6825/2717758054","cluster_addr":"172.21.15.110:6827/2717758054","heartbeat_back_addr":"172.21.15.110:6831/2717758054","heartbeat_front_addr":"172.21.15.110:6829/2717758054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:51:31.165874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:51:57.295234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:52:23.000112+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:52:49.157367+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:53:13.758729+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:53:40.969407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:54:08.014278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:54:35.396853+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2821019614":"2024-03-12T21:50:12.284168+0000","172.21.15.102:0/3686793380":"2024-03-12T21:50:12.284168+0000","172.21.15.102:0/432123271":"2024-03-12T21:50:12.284168+0000","172.21.15.102:6801/2761823548":"2024-03-12T21:50:12.284168+0000","172.21.15.102:6801/161836558":"2024-03-12T21:49:47.015350+0000","172.21.15.102:0/3333959561":"2024-03-12T21:49:47.015350+0000","172.21.15.102:6800/2761823548":"2024-03-12T21:50:12.284168+0000","172.21.15.102:6800/161836558":"2024-03-12T21:49:47.015350+0000","172.21.15.102:0/4278625363":"2024-03-12T21:49:47.015350+0000","172.21.15.102:0/3679135738":"2024-03-12T21:49:47.015350+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:55:28.896 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:28 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1821301659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:55:28.896 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:28 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:28.615+0000 7ff2f1a79200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:55:28.896 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:28 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:28.705+0000 7ff2f1a79200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:55:28.896 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:28 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:28.902+0000 7ff2f1a79200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:55:29.056 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:28 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1821301659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:55:29.057 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:28 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1821301659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:55:29.057 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:28 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:28.771+0000 7f7eeeef0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:55:29.058 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:28 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:28.864+0000 7f7eeeef0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:55:29.058 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:29 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:29.063+0000 7f7eeeef0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:55:29.238 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:29 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:29.086+0000 7ff2f1a79200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:55:29.349 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:29 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:29.250+0000 7f7eeeef0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:55:29.377 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T21:55:29.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:55:29.618 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:29 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:29.368+0000 7ff2f1a79200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:55:29.618 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:29 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:29.625+0000 7ff2f1a79200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:55:29.621 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:29.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:29 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:29.551+0000 7f7eeeef0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:55:29.988 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:29 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:29.863+0000 7ff2f1a79200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:55:30.064 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:29 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:29.822+0000 7f7eeeef0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:55:30.064 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:30 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:30.070+0000 7f7eeeef0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:55:30.988 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:30 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:30.663+0000 7ff2f1a79200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:55:31.010 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:30 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:30.891+0000 7f7eeeef0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:55:31.479 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:55:31.479 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":71,"fsid":"0126e6b2-dff1-11ee-95c8-87774f69a715","created":"2024-03-11T21:49:14.830092+0000","modified":"2024-03-11T21:55:13.495656+0000","last_up_change":"2024-03-11T21:54:36.635837+0000","last_in_change":"2024-03-11T21:54:18.389315+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:52:26.615067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-11T21:54:48.886488+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-11T21:54:49.119968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:54:50.734942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:54:52.673627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:54:54.747224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1f36f4f8-fe18-4d44-b3df-b07709f4af70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6803","nonce":1026485852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6805","nonce":1026485852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6809","nonce":1026485852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1026485852},{"type":"v1","addr":"172.21.15.102:6807","nonce":1026485852}]},"public_addr":"172.21.15.102:6803/1026485852","cluster_addr":"172.21.15.102:6805/1026485852","heartbeat_back_addr":"172.21.15.102:6809/1026485852","heartbeat_front_addr":"172.21.15.102:6807/1026485852","state":["exists","up"]},{"osd":1,"uuid":"9bd35249-7d61-4d04-ba14-31debbf69fd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6811","nonce":265722643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6813","nonce":265722643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6817","nonce":265722643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":265722643},{"type":"v1","addr":"172.21.15.102:6815","nonce":265722643}]},"public_addr":"172.21.15.102:6811/265722643","cluster_addr":"172.21.15.102:6813/265722643","heartbeat_back_addr":"172.21.15.102:6817/265722643","heartbeat_front_addr":"172.21.15.102:6815/265722643","state":["exists","up"]},{"osd":2,"uuid":"6c25d672-a2ee-420d-9008-cb48d85c3b57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6819","nonce":2501373355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6821","nonce":2501373355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6825","nonce":2501373355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2501373355},{"type":"v1","addr":"172.21.15.102:6823","nonce":2501373355}]},"public_addr":"172.21.15.102:6819/2501373355","cluster_addr":"172.21.15.102:6821/2501373355","heartbeat_back_addr":"172.21.15.102:6825/2501373355","heartbeat_front_addr":"172.21.15.102:6823/2501373355","state":["exists","up"]},{"osd":3,"uuid":"89807137-de78-4a04-8442-478a3e2eb242","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6827","nonce":1356933888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6829","nonce":1356933888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6833","nonce":1356933888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1356933888},{"type":"v1","addr":"172.21.15.102:6831","nonce":1356933888}]},"public_addr":"172.21.15.102:6827/1356933888","cluster_addr":"172.21.15.102:6829/1356933888","heartbeat_back_addr":"172.21.15.102:6833/1356933888","heartbeat_front_addr":"172.21.15.102:6831/1356933888","state":["exists","up"]},{"osd":4,"uuid":"c670cc8e-51e7-4b4c-8e20-160b54ab7688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6801","nonce":882287460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6803","nonce":882287460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6807","nonce":882287460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":882287460},{"type":"v1","addr":"172.21.15.110:6805","nonce":882287460}]},"public_addr":"172.21.15.110:6801/882287460","cluster_addr":"172.21.15.110:6803/882287460","heartbeat_back_addr":"172.21.15.110:6807/882287460","heartbeat_front_addr":"172.21.15.110:6805/882287460","state":["exists","up"]},{"osd":5,"uuid":"e087753a-eddf-4880-a51e-8efdfec06331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6809","nonce":1539634603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6811","nonce":1539634603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6815","nonce":1539634603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1539634603},{"type":"v1","addr":"172.21.15.110:6813","nonce":1539634603}]},"public_addr":"172.21.15.110:6809/1539634603","cluster_addr":"172.21.15.110:6811/1539634603","heartbeat_back_addr":"172.21.15.110:6815/1539634603","heartbeat_front_addr":"172.21.15.110:6813/1539634603","state":["exists","up"]},{"osd":6,"uuid":"e599016e-0eeb-4fdd-a8a9-6fc8eed1218a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6817","nonce":462580487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6819","nonce":462580487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6823","nonce":462580487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462580487},{"type":"v1","addr":"172.21.15.110:6821","nonce":462580487}]},"public_addr":"172.21.15.110:6817/462580487","cluster_addr":"172.21.15.110:6819/462580487","heartbeat_back_addr":"172.21.15.110:6823/462580487","heartbeat_front_addr":"172.21.15.110:6821/462580487","state":["exists","up"]},{"osd":7,"uuid":"800b08ef-0293-491b-a615-0ab0692fd14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6825","nonce":2717758054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6827","nonce":2717758054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6831","nonce":2717758054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2717758054},{"type":"v1","addr":"172.21.15.110:6829","nonce":2717758054}]},"public_addr":"172.21.15.110:6825/2717758054","cluster_addr":"172.21.15.110:6827/2717758054","heartbeat_back_addr":"172.21.15.110:6831/2717758054","heartbeat_front_addr":"172.21.15.110:6829/2717758054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:51:31.165874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:51:57.295234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:52:23.000112+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:52:49.157367+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:53:13.758729+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:53:40.969407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:54:08.014278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:54:35.396853+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2821019614":"2024-03-12T21:50:12.284168+0000","172.21.15.102:0/3686793380":"2024-03-12T21:50:12.284168+0000","172.21.15.102:0/432123271":"2024-03-12T21:50:12.284168+0000","172.21.15.102:6801/2761823548":"2024-03-12T21:50:12.284168+0000","172.21.15.102:6801/161836558":"2024-03-12T21:49:47.015350+0000","172.21.15.102:0/3333959561":"2024-03-12T21:49:47.015350+0000","172.21.15.102:6800/2761823548":"2024-03-12T21:50:12.284168+0000","172.21.15.102:6800/161836558":"2024-03-12T21:49:47.015350+0000","172.21.15.102:0/4278625363":"2024-03-12T21:49:47.015350+0000","172.21.15.102:0/3679135738":"2024-03-12T21:49:47.015350+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:55:31.488 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:31.230+0000 7ff2f1a79200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:55:31.488 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:31.329+0000 7ff2f1a79200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:55:31.805 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:31 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/75329762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:55:31.806 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:31 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/75329762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:55:31.806 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:31 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:31.482+0000 7f7eeeef0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:55:31.806 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:31 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:31.590+0000 7f7eeeef0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:55:31.806 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:31 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:31.812+0000 7f7eeeef0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:55:31.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/75329762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:55:31.942 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:31.545+0000 7ff2f1a79200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:55:31.943 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: [11/Mar/2024:21:55:31] ENGINE Bus STARTING 2024-03-11T21:55:31.943 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: CherryPy Checker: 2024-03-11T21:55:31.943 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: The Application mounted at '' has an empty config. 2024-03-11T21:55:31.943 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: 2024-03-11T21:55:31.943 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: [11/Mar/2024:21:55:31] ENGINE Serving on http://:::9283 2024-03-11T21:55:31.943 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:55:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x[32413]: [11/Mar/2024:21:55:31] ENGINE Bus STARTED 2024-03-11T21:55:32.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:31 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:55:31] ENGINE Bus STARTING 2024-03-11T21:55:32.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: CherryPy Checker: 2024-03-11T21:55:32.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: The Application mounted at '' has an empty config. 2024-03-11T21:55:32.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:55:32.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:55:32] ENGINE Serving on http://:::9283 2024-03-11T21:55:32.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:55:32] ENGINE Bus STARTED 2024-03-11T21:55:32.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T21:55:32.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T21:55:32.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T21:55:32.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T21:55:32.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T21:55:32.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T21:55:32.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T21:55:32.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T21:55:32.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: Standby manager daemon x restarted 2024-03-11T21:55:32.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: Standby manager daemon x started 2024-03-11T21:55:32.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:55:32.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:55:32.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:55:32.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:55:32.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: Active manager daemon y restarted 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: Activating manager daemon y 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: mgrmap e18: y(active, starting, since 0.00789358s), standbys: x 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:55:32.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: Manager daemon y is now available 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:55:32.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:55:32.538 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:55:32.765 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:32.773 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:32.846 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: Standby manager daemon x restarted 2024-03-11T21:55:32.846 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: Standby manager daemon x started 2024-03-11T21:55:32.846 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: Active manager daemon y restarted 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: Activating manager daemon y 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: mgrmap e18: y(active, starting, since 0.00789358s), standbys: x 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:55:32.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: Manager daemon y is now available 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:55:32.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:55:32.849 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:55:32.914 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: Standby manager daemon x restarted 2024-03-11T21:55:32.914 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: Standby manager daemon x started 2024-03-11T21:55:32.914 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:55:32.914 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:55:32.914 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.? 172.21.15.110:0/2962390533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: Active manager daemon y restarted 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: Activating manager daemon y 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: mgrmap e18: y(active, starting, since 0.00789358s), standbys: x 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:55:32.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: Manager daemon y is now available 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:55:32.916 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:55:33.127 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:32 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:55:34.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:33 smithi102 ceph-mon[25338]: mgrmap e19: y(active, since 1.03205s), standbys: x 2024-03-11T21:55:34.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:33 smithi102 ceph-mon[34177]: mgrmap e19: y(active, since 1.03205s), standbys: x 2024-03-11T21:55:34.206 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:34.207 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:34.207 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:34.209 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:34.209 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:34.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:33 smithi110 ceph-mon[30535]: mgrmap e19: y(active, since 1.03205s), standbys: x 2024-03-11T21:55:35.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[25338]: [11/Mar/2024:21:55:33] ENGINE Bus STARTING 2024-03-11T21:55:35.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[25338]: [11/Mar/2024:21:55:33] ENGINE Serving on https://172.21.15.102:7150 2024-03-11T21:55:35.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[25338]: [11/Mar/2024:21:55:33] ENGINE Serving on http://172.21.15.102:8765 2024-03-11T21:55:35.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[25338]: [11/Mar/2024:21:55:33] ENGINE Bus STARTED 2024-03-11T21:55:35.099 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[25338]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:55:35.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[34177]: [11/Mar/2024:21:55:33] ENGINE Bus STARTING 2024-03-11T21:55:35.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[34177]: [11/Mar/2024:21:55:33] ENGINE Serving on https://172.21.15.102:7150 2024-03-11T21:55:35.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[34177]: [11/Mar/2024:21:55:33] ENGINE Serving on http://172.21.15.102:8765 2024-03-11T21:55:35.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[34177]: [11/Mar/2024:21:55:33] ENGINE Bus STARTED 2024-03-11T21:55:35.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:34 smithi102 ceph-mon[34177]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:55:35.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:34 smithi110 ceph-mon[30535]: [11/Mar/2024:21:55:33] ENGINE Bus STARTING 2024-03-11T21:55:35.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:34 smithi110 ceph-mon[30535]: [11/Mar/2024:21:55:33] ENGINE Serving on https://172.21.15.102:7150 2024-03-11T21:55:35.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:34 smithi110 ceph-mon[30535]: [11/Mar/2024:21:55:33] ENGINE Serving on http://172.21.15.102:8765 2024-03-11T21:55:35.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:34 smithi110 ceph-mon[30535]: [11/Mar/2024:21:55:33] ENGINE Bus STARTED 2024-03-11T21:55:35.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:34 smithi110 ceph-mon[30535]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:55:36.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:35 smithi102 ceph-mon[25338]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-11T21:55:36.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:35 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:36.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:35 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:36.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:35 smithi102 ceph-mon[25338]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:55:36.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:35 smithi102 ceph-mon[34177]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-11T21:55:36.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:35 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:36.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:35 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:36.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:35 smithi102 ceph-mon[34177]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:55:36.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:35 smithi110 ceph-mon[30535]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-11T21:55:36.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:35 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:36.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:35 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:36.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:35 smithi110 ceph-mon[30535]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:55:37.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[25338]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T21:55:37.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:37.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:37.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi110 to 3760M 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[34177]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi110 to 3760M 2024-03-11T21:55:37.259 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:37 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:37 smithi110 ceph-mon[30535]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T21:55:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:37 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:37.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:37 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:37.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:37 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:37 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:37 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:37 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:55:37.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:37 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi110 to 3760M 2024-03-11T21:55:37.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:37 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:55:38.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:38 smithi102 ceph-mon[25338]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:55:38.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:38 smithi102 ceph-mon[34177]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:55:38.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:38 smithi110 ceph-mon[30535]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:55:40.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:39 smithi102 ceph-mon[25338]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T21:55:40.052 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:39 smithi102 ceph-mon[34177]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T21:55:40.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:39 smithi110 ceph-mon[30535]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T21:55:40.388 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:40 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ::ffff:172.21.15.110 - - [11/Mar/2024:21:55:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:55:41.855 INFO:teuthology.orchestra.run.smithi102.stdout:55834574894 2024-03-11T21:55:41.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T21:55:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:41 smithi102 ceph-mon[25338]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T21:55:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:41 smithi102 ceph-mon[34177]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T21:55:42.227 INFO:teuthology.orchestra.run.smithi102.stdout:77309411369 2024-03-11T21:55:42.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T21:55:42.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:41 smithi110 ceph-mon[30535]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T21:55:43.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:55:43.682 INFO:teuthology.orchestra.run.smithi102.stdout:133143986206 2024-03-11T21:55:43.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T21:55:43.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:43 smithi102 ceph-mon[25338]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:43.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:43 smithi102 ceph-mon[25338]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T21:55:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:43 smithi102 ceph-mon[34177]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:43 smithi102 ceph-mon[34177]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T21:55:44.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:43 smithi110 ceph-mon[30535]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:44.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:43 smithi110 ceph-mon[30535]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T21:55:45.627 INFO:teuthology.orchestra.run.smithi102.stdout:167503724570 2024-03-11T21:55:45.627 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T21:55:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:45 smithi102 ceph-mon[25338]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:55:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:45 smithi102 ceph-mon[34177]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:55:46.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:45 smithi110 ceph-mon[30535]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:55:47.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:46 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:55:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:46 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:55:47.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:46 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:55:47.635 INFO:teuthology.orchestra.run.smithi102.stdout:34359738421 2024-03-11T21:55:47.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T21:55:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:48 smithi102 ceph-mon[25338]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:55:48.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:48 smithi102 ceph-mon[34177]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:55:48.318 INFO:teuthology.orchestra.run.smithi102.stdout:236223201295 2024-03-11T21:55:48.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T21:55:48.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:48 smithi110 ceph-mon[30535]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:55:50.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:49 smithi102 ceph-mon[25338]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:55:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:49 smithi102 ceph-mon[34177]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:55:50.218 INFO:teuthology.orchestra.run.smithi102.stdout:201863462932 2024-03-11T21:55:50.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T21:55:50.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:49 smithi110 ceph-mon[30535]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:55:50.663 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:55:50 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ::ffff:172.21.15.110 - - [11/Mar/2024:21:55:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:55:50.697 INFO:teuthology.orchestra.run.smithi102.stdout:111669149734 2024-03-11T21:55:50.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T21:55:51.352 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:51.352 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:51.353 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:51.646 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:51.647 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:52.072 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:51 smithi102 ceph-mon[25338]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:55:52.072 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:51 smithi102 ceph-mon[34177]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:55:52.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:51 smithi110 ceph-mon[30535]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:55:52.756 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:53.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:55:52 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:55:54.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:53 smithi102 ceph-mon[25338]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:54.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:53 smithi102 ceph-mon[25338]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:55:54.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:53 smithi102 ceph-mon[34177]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:54.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:53 smithi102 ceph-mon[34177]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:55:54.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:53 smithi110 ceph-mon[30535]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:55:54.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:53 smithi110 ceph-mon[30535]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:55:56.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:55 smithi102 ceph-mon[25338]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:55:56.111 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:55 smithi102 ceph-mon[34177]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:55:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:55:56.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:55 smithi110 ceph-mon[30535]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:55:58.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:57 smithi102 ceph-mon[25338]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:55:58.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:57 smithi102 ceph-mon[34177]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:55:58.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:57 smithi110 ceph-mon[30535]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout:55834574897 2024-03-11T21:55:59.802 INFO:teuthology.orchestra.run.smithi102.stdout:111669149735 2024-03-11T21:56:00.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:59 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:00.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:59 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:00.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:59 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/118910534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:56:00.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:59 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/3067016374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:56:00.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:55:59 smithi110 ceph-mon[30535]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-11T21:56:00.341 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:00 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ::ffff:172.21.15.110 - - [11/Mar/2024:21:56:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:56:00.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:00.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:00.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/118910534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:56:00.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/3067016374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:56:00.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[25338]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-11T21:56:00.342 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:00.342 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:00.342 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/118910534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:56:00.342 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/3067016374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:56:00.342 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:55:59 smithi102 ceph-mon[34177]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: Adjusting osd_memory_target on smithi102 to 3760M 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:02.533 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: Updating smithi110:/etc/ceph/ceph.conf 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.534 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: Adjusting osd_memory_target on smithi102 to 3760M 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: Updating smithi110:/etc/ceph/ceph.conf 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:02.535 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:02.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:02.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.536 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:02 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: Adjusting osd_memory_target on smithi102 to 3760M 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:02.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: Updating smithi110:/etc/ceph/ceph.conf 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:02.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:03.030 INFO:teuthology.orchestra.run.smithi102.stdout:77309411373 2024-03-11T21:56:03.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:56:02 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:56:03.532 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574897 for osd.1 2024-03-11T21:56:03.532 DEBUG:teuthology.parallel:result is None 2024-03-11T21:56:03.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[25338]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[25338]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[25338]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[25338]: Deploying daemon alertmanager.a on smithi102 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1179039983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1353880363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[34177]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[34177]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[34177]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[34177]: Deploying daemon alertmanager.a on smithi102 2024-03-11T21:56:03.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1179039983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:56:03.636 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:03 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1353880363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:56:03.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:03 smithi110 ceph-mon[30535]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:03.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:03 smithi110 ceph-mon[30535]: Updating smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:56:03.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:03 smithi110 ceph-mon[30535]: Updating smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:56:03.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:03 smithi110 ceph-mon[30535]: Deploying daemon alertmanager.a on smithi102 2024-03-11T21:56:03.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:03 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1179039983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:56:03.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:03 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1353880363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:56:04.374 INFO:teuthology.orchestra.run.smithi102.stdout:133143986210 2024-03-11T21:56:04.583 INFO:teuthology.orchestra.run.smithi102.stdout:236223201299 2024-03-11T21:56:04.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:04 smithi102 ceph-mon[25338]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:04.599 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:04 smithi102 ceph-mon[25338]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:04.599 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:04 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1693619754' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:56:04.599 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:04 smithi102 ceph-mon[34177]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:04.599 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:04 smithi102 ceph-mon[34177]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:04.599 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:04 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1693619754' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:56:04.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:04 smithi110 ceph-mon[30535]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:04.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:04 smithi110 ceph-mon[30535]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:04.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:04 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1693619754' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:56:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:05 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1180653238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:56:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:05 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1180653238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:56:05.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:05 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1180653238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:56:05.866 INFO:teuthology.orchestra.run.smithi102.stdout:167503724573 2024-03-11T21:56:05.934 INFO:teuthology.orchestra.run.smithi102.stdout:201863462935 2024-03-11T21:56:06.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:06 smithi102 ceph-mon[25338]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:06.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:06 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/537898170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:56:06.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:06 smithi102 ceph-mon[34177]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:06.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:06 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/537898170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:56:06.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:06 smithi110 ceph-mon[30535]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:06.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:06 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/537898170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:56:07.220 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201295 got 236223201299 for osd.7 2024-03-11T21:56:07.220 DEBUG:teuthology.parallel:result is None 2024-03-11T21:56:07.529 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411373 for osd.2 2024-03-11T21:56:07.529 DEBUG:teuthology.parallel:result is None 2024-03-11T21:56:07.739 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724570 got 167503724573 for osd.5 2024-03-11T21:56:07.739 DEBUG:teuthology.parallel:result is None 2024-03-11T21:56:08.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:07 smithi102 ceph-mon[25338]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:08.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:07 smithi102 ceph-mon[34177]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:08.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:07 smithi110 ceph-mon[30535]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:08.479 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986206 got 133143986210 for osd.4 2024-03-11T21:56:08.480 DEBUG:teuthology.parallel:result is None 2024-03-11T21:56:08.908 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149734 got 111669149735 for osd.3 2024-03-11T21:56:08.908 DEBUG:teuthology.parallel:result is None 2024-03-11T21:56:09.341 INFO:teuthology.orchestra.run.smithi102.stdout:34359738425 2024-03-11T21:56:09.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:09 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/2007877016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:56:09.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:09 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/2007877016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:56:09.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:09 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/2007877016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:56:10.401 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:10 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ::ffff:172.21.15.110 - - [11/Mar/2024:21:56:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:56:10.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738421 got 34359738425 for osd.0 2024-03-11T21:56:10.552 DEBUG:teuthology.parallel:result is None 2024-03-11T21:56:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:10 smithi102 ceph-mon[25338]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:10 smithi102 ceph-mon[34177]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:10.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:10 smithi110 ceph-mon[30535]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:10.879 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462932 got 201863462935 for osd.6 2024-03-11T21:56:10.879 DEBUG:teuthology.parallel:result is None 2024-03-11T21:56:10.879 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T21:56:10.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:56:11.171 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:11.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:11 smithi102 ceph-mon[25338]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:11.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:11 smithi102 ceph-mon[34177]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:12.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:11 smithi110 ceph-mon[30535]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:13.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:56:12 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:56:13.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:13 smithi102 ceph-mon[25338]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:13.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:13 smithi102 ceph-mon[25338]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:13.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:13 smithi102 ceph-mon[34177]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:13.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:13 smithi102 ceph-mon[34177]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:14.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:13 smithi110 ceph-mon[30535]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:14.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:13 smithi110 ceph-mon[30535]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:16.076 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:56:16.076 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:15 smithi102 ceph-mon[25338]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:16.077 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:15 smithi102 ceph-mon[34177]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:16.083 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-11T21:56:16.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:15 smithi110 ceph-mon[30535]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:16.377 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:16 smithi102 systemd[1]: Starting Ceph alertmanager.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:56:16.796 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-11T21:56:15.849140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1039,"num_read_kb":882,"num_write":681,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":661,"ondisk_log_size":661,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":236700,"kb_used_data":8060,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749691748,"statfs":{"total":767926730752,"available":767684349952,"internally_reserved":0,"allocated":8253440,"data_stored":4456524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002779"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831881+0000","last_change":"2024-03-11T21:54:56.449478+0000","last_active":"2024-03-11T21:55:31.831881+0000","last_peered":"2024-03-11T21:55:31.831881+0000","last_clean":"2024-03-11T21:55:31.831881+0000","last_became_active":"2024-03-11T21:54:56.449175+0000","last_became_peered":"2024-03-11T21:54:56.449175+0000","last_unstale":"2024-03-11T21:55:31.831881+0000","last_undegraded":"2024-03-11T21:55:31.831881+0000","last_fullsized":"2024-03-11T21:55:31.831881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:42:38.581508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830654+0000","last_change":"2024-03-11T21:54:54.449954+0000","last_active":"2024-03-11T21:55:31.830654+0000","last_peered":"2024-03-11T21:55:31.830654+0000","last_clean":"2024-03-11T21:55:31.830654+0000","last_became_active":"2024-03-11T21:54:54.449654+0000","last_became_peered":"2024-03-11T21:54:54.449654+0000","last_unstale":"2024-03-11T21:55:31.830654+0000","last_undegraded":"2024-03-11T21:55:31.830654+0000","last_fullsized":"2024-03-11T21:55:31.830654+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:35.109872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065812+0000","last_change":"2024-03-11T21:54:50.419611+0000","last_active":"2024-03-11T21:55:32.065812+0000","last_peered":"2024-03-11T21:55:32.065812+0000","last_clean":"2024-03-11T21:55:32.065812+0000","last_became_active":"2024-03-11T21:54:50.419123+0000","last_became_peered":"2024-03-11T21:54:50.419123+0000","last_unstale":"2024-03-11T21:55:32.065812+0000","last_undegraded":"2024-03-11T21:55:32.065812+0000","last_fullsized":"2024-03-11T21:55:32.065812+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:41:49.213574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832038+0000","last_change":"2024-03-11T21:55:13.514481+0000","last_active":"2024-03-11T21:55:31.832038+0000","last_peered":"2024-03-11T21:55:31.832038+0000","last_clean":"2024-03-11T21:55:31.832038+0000","last_became_active":"2024-03-11T21:55:13.508420+0000","last_became_peered":"2024-03-11T21:55:13.508420+0000","last_unstale":"2024-03-11T21:55:31.832038+0000","last_undegraded":"2024-03-11T21:55:31.832038+0000","last_fullsized":"2024-03-11T21:55:31.832038+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:21:57.498731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065699+0000","last_change":"2024-03-11T21:54:56.537559+0000","last_active":"2024-03-11T21:55:32.065699+0000","last_peered":"2024-03-11T21:55:32.065699+0000","last_clean":"2024-03-11T21:55:32.065699+0000","last_became_active":"2024-03-11T21:54:56.537217+0000","last_became_peered":"2024-03-11T21:54:56.537217+0000","last_unstale":"2024-03-11T21:55:32.065699+0000","last_undegraded":"2024-03-11T21:55:32.065699+0000","last_fullsized":"2024-03-11T21:55:32.065699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:53:16.201491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830944+0000","last_change":"2024-03-11T21:54:54.445651+0000","last_active":"2024-03-11T21:55:31.830944+0000","last_peered":"2024-03-11T21:55:31.830944+0000","last_clean":"2024-03-11T21:55:31.830944+0000","last_became_active":"2024-03-11T21:54:54.445342+0000","last_became_peered":"2024-03-11T21:54:54.445342+0000","last_unstale":"2024-03-11T21:55:31.830944+0000","last_undegraded":"2024-03-11T21:55:31.830944+0000","last_fullsized":"2024-03-11T21:55:31.830944+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:15:59.165041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066715+0000","last_change":"2024-03-11T21:54:50.413129+0000","last_active":"2024-03-11T21:55:32.066715+0000","last_peered":"2024-03-11T21:55:32.066715+0000","last_clean":"2024-03-11T21:55:32.066715+0000","last_became_active":"2024-03-11T21:54:50.412818+0000","last_became_peered":"2024-03-11T21:54:50.412818+0000","last_unstale":"2024-03-11T21:55:32.066715+0000","last_undegraded":"2024-03-11T21:55:32.066715+0000","last_fullsized":"2024-03-11T21:55:32.066715+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:10.340260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065828+0000","last_change":"2024-03-11T21:54:52.438782+0000","last_active":"2024-03-11T21:55:32.065828+0000","last_peered":"2024-03-11T21:55:32.065828+0000","last_clean":"2024-03-11T21:55:32.065828+0000","last_became_active":"2024-03-11T21:54:52.438472+0000","last_became_peered":"2024-03-11T21:54:52.438472+0000","last_unstale":"2024-03-11T21:55:32.065828+0000","last_undegraded":"2024-03-11T21:55:32.065828+0000","last_fullsized":"2024-03-11T21:55:32.065828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:43.708792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065986+0000","last_change":"2024-03-11T21:54:56.456252+0000","last_active":"2024-03-11T21:55:32.065986+0000","last_peered":"2024-03-11T21:55:32.065986+0000","last_clean":"2024-03-11T21:55:32.065986+0000","last_became_active":"2024-03-11T21:54:56.455938+0000","last_became_peered":"2024-03-11T21:54:56.455938+0000","last_unstale":"2024-03-11T21:55:32.065986+0000","last_undegraded":"2024-03-11T21:55:32.065986+0000","last_fullsized":"2024-03-11T21:55:32.065986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:38:27.060415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933966+0000","last_change":"2024-03-11T21:54:54.455992+0000","last_active":"2024-03-11T21:55:31.933966+0000","last_peered":"2024-03-11T21:55:31.933966+0000","last_clean":"2024-03-11T21:55:31.933966+0000","last_became_active":"2024-03-11T21:54:54.455634+0000","last_became_peered":"2024-03-11T21:54:54.455634+0000","last_unstale":"2024-03-11T21:55:31.933966+0000","last_undegraded":"2024-03-11T21:55:31.933966+0000","last_fullsized":"2024-03-11T21:55:31.933966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:45.875406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934060+0000","last_change":"2024-03-11T21:54:50.418368+0000","last_active":"2024-03-11T21:55:31.934060+0000","last_peered":"2024-03-11T21:55:31.934060+0000","last_clean":"2024-03-11T21:55:31.934060+0000","last_became_active":"2024-03-11T21:54:50.418011+0000","last_became_peered":"2024-03-11T21:54:50.418011+0000","last_unstale":"2024-03-11T21:55:31.934060+0000","last_undegraded":"2024-03-11T21:55:31.934060+0000","last_fullsized":"2024-03-11T21:55:31.934060+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:03.100681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832113+0000","last_change":"2024-03-11T21:54:52.433188+0000","last_active":"2024-03-11T21:55:31.832113+0000","last_peered":"2024-03-11T21:55:31.832113+0000","last_clean":"2024-03-11T21:55:31.832113+0000","last_became_active":"2024-03-11T21:54:52.432859+0000","last_became_peered":"2024-03-11T21:54:52.432859+0000","last_unstale":"2024-03-11T21:55:31.832113+0000","last_undegraded":"2024-03-11T21:55:31.832113+0000","last_fullsized":"2024-03-11T21:55:31.832113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:15:39.303127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933699+0000","last_change":"2024-03-11T21:54:56.463176+0000","last_active":"2024-03-11T21:55:31.933699+0000","last_peered":"2024-03-11T21:55:31.933699+0000","last_clean":"2024-03-11T21:55:31.933699+0000","last_became_active":"2024-03-11T21:54:56.462892+0000","last_became_peered":"2024-03-11T21:54:56.462892+0000","last_unstale":"2024-03-11T21:55:31.933699+0000","last_undegraded":"2024-03-11T21:55:31.933699+0000","last_fullsized":"2024-03-11T21:55:31.933699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:31.708144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946369+0000","last_change":"2024-03-11T21:54:54.455379+0000","last_active":"2024-03-11T21:55:31.946369+0000","last_peered":"2024-03-11T21:55:31.946369+0000","last_clean":"2024-03-11T21:55:31.946369+0000","last_became_active":"2024-03-11T21:54:54.455073+0000","last_became_peered":"2024-03-11T21:54:54.455073+0000","last_unstale":"2024-03-11T21:55:31.946369+0000","last_undegraded":"2024-03-11T21:55:31.946369+0000","last_fullsized":"2024-03-11T21:55:31.946369+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:30.235270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934133+0000","last_change":"2024-03-11T21:54:50.410817+0000","last_active":"2024-03-11T21:55:31.934133+0000","last_peered":"2024-03-11T21:55:31.934133+0000","last_clean":"2024-03-11T21:55:31.934133+0000","last_became_active":"2024-03-11T21:54:50.410241+0000","last_became_peered":"2024-03-11T21:54:50.410241+0000","last_unstale":"2024-03-11T21:55:31.934133+0000","last_undegraded":"2024-03-11T21:55:31.934133+0000","last_fullsized":"2024-03-11T21:55:31.934133+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:49.365681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831637+0000","last_change":"2024-03-11T21:54:52.438105+0000","last_active":"2024-03-11T21:55:31.831637+0000","last_peered":"2024-03-11T21:55:31.831637+0000","last_clean":"2024-03-11T21:55:31.831637+0000","last_became_active":"2024-03-11T21:54:52.437744+0000","last_became_peered":"2024-03-11T21:54:52.437744+0000","last_unstale":"2024-03-11T21:55:31.831637+0000","last_undegraded":"2024-03-11T21:55:31.831637+0000","last_fullsized":"2024-03-11T21:55:31.831637+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:37.745150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831463+0000","last_change":"2024-03-11T21:54:56.537012+0000","last_active":"2024-03-11T21:55:31.831463+0000","last_peered":"2024-03-11T21:55:31.831463+0000","last_clean":"2024-03-11T21:55:31.831463+0000","last_became_active":"2024-03-11T21:54:56.536644+0000","last_became_peered":"2024-03-11T21:54:56.536644+0000","last_unstale":"2024-03-11T21:55:31.831463+0000","last_undegraded":"2024-03-11T21:55:31.831463+0000","last_fullsized":"2024-03-11T21:55:31.831463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:57.627059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831565+0000","last_change":"2024-03-11T21:54:54.445955+0000","last_active":"2024-03-11T21:55:31.831565+0000","last_peered":"2024-03-11T21:55:31.831565+0000","last_clean":"2024-03-11T21:55:31.831565+0000","last_became_active":"2024-03-11T21:54:54.445624+0000","last_became_peered":"2024-03-11T21:54:54.445624+0000","last_unstale":"2024-03-11T21:55:31.831565+0000","last_undegraded":"2024-03-11T21:55:31.831565+0000","last_fullsized":"2024-03-11T21:55:31.831565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:57:38.628568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066562+0000","last_change":"2024-03-11T21:54:50.422131+0000","last_active":"2024-03-11T21:55:32.066562+0000","last_peered":"2024-03-11T21:55:32.066562+0000","last_clean":"2024-03-11T21:55:32.066562+0000","last_became_active":"2024-03-11T21:54:50.421356+0000","last_became_peered":"2024-03-11T21:54:50.421356+0000","last_unstale":"2024-03-11T21:55:32.066562+0000","last_undegraded":"2024-03-11T21:55:32.066562+0000","last_fullsized":"2024-03-11T21:55:32.066562+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:14:21.895413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832590+0000","last_change":"2024-03-11T21:54:52.432518+0000","last_active":"2024-03-11T21:55:31.832590+0000","last_peered":"2024-03-11T21:55:31.832590+0000","last_clean":"2024-03-11T21:55:31.832590+0000","last_became_active":"2024-03-11T21:54:52.432140+0000","last_became_peered":"2024-03-11T21:54:52.432140+0000","last_unstale":"2024-03-11T21:55:31.832590+0000","last_undegraded":"2024-03-11T21:55:31.832590+0000","last_fullsized":"2024-03-11T21:55:31.832590+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:51:11.877624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066529+0000","last_change":"2024-03-11T21:54:56.536166+0000","last_active":"2024-03-11T21:55:32.066529+0000","last_peered":"2024-03-11T21:55:32.066529+0000","last_clean":"2024-03-11T21:55:32.066529+0000","last_became_active":"2024-03-11T21:54:56.535015+0000","last_became_peered":"2024-03-11T21:54:56.535015+0000","last_unstale":"2024-03-11T21:55:32.066529+0000","last_undegraded":"2024-03-11T21:55:32.066529+0000","last_fullsized":"2024-03-11T21:55:32.066529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:02:49.161746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831471+0000","last_change":"2024-03-11T21:54:54.453732+0000","last_active":"2024-03-11T21:55:31.831471+0000","last_peered":"2024-03-11T21:55:31.831471+0000","last_clean":"2024-03-11T21:55:31.831471+0000","last_became_active":"2024-03-11T21:54:54.453333+0000","last_became_peered":"2024-03-11T21:54:54.453333+0000","last_unstale":"2024-03-11T21:55:31.831471+0000","last_undegraded":"2024-03-11T21:55:31.831471+0000","last_fullsized":"2024-03-11T21:55:31.831471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:34.135236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832230+0000","last_change":"2024-03-11T21:54:50.412648+0000","last_active":"2024-03-11T21:55:31.832230+0000","last_peered":"2024-03-11T21:55:31.832230+0000","last_clean":"2024-03-11T21:55:31.832230+0000","last_became_active":"2024-03-11T21:54:50.410393+0000","last_became_peered":"2024-03-11T21:54:50.410393+0000","last_unstale":"2024-03-11T21:55:31.832230+0000","last_undegraded":"2024-03-11T21:55:31.832230+0000","last_fullsized":"2024-03-11T21:55:31.832230+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:41:50.324281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066283+0000","last_change":"2024-03-11T21:54:52.436246+0000","last_active":"2024-03-11T21:55:32.066283+0000","last_peered":"2024-03-11T21:55:32.066283+0000","last_clean":"2024-03-11T21:55:32.066283+0000","last_became_active":"2024-03-11T21:54:52.434955+0000","last_became_peered":"2024-03-11T21:54:52.434955+0000","last_unstale":"2024-03-11T21:55:32.066283+0000","last_undegraded":"2024-03-11T21:55:32.066283+0000","last_fullsized":"2024-03-11T21:55:32.066283+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:37:26.945043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946549+0000","last_change":"2024-03-11T21:54:56.456693+0000","last_active":"2024-03-11T21:55:31.946549+0000","last_peered":"2024-03-11T21:55:31.946549+0000","last_clean":"2024-03-11T21:55:31.946549+0000","last_became_active":"2024-03-11T21:54:56.456371+0000","last_became_peered":"2024-03-11T21:54:56.456371+0000","last_unstale":"2024-03-11T21:55:31.946549+0000","last_undegraded":"2024-03-11T21:55:31.946549+0000","last_fullsized":"2024-03-11T21:55:31.946549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:24:41.760797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472140+0000","last_change":"2024-03-11T21:54:54.448752+0000","last_active":"2024-03-11T21:56:05.472140+0000","last_peered":"2024-03-11T21:56:05.472140+0000","last_clean":"2024-03-11T21:56:05.472140+0000","last_became_active":"2024-03-11T21:54:54.448398+0000","last_became_peered":"2024-03-11T21:54:54.448398+0000","last_unstale":"2024-03-11T21:56:05.472140+0000","last_undegraded":"2024-03-11T21:56:05.472140+0000","last_fullsized":"2024-03-11T21:56:05.472140+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:03:20.410444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830787+0000","last_change":"2024-03-11T21:54:50.412242+0000","last_active":"2024-03-11T21:55:31.830787+0000","last_peered":"2024-03-11T21:55:31.830787+0000","last_clean":"2024-03-11T21:55:31.830787+0000","last_became_active":"2024-03-11T21:54:50.411916+0000","last_became_peered":"2024-03-11T21:54:50.411916+0000","last_unstale":"2024-03-11T21:55:31.830787+0000","last_undegraded":"2024-03-11T21:55:31.830787+0000","last_fullsized":"2024-03-11T21:55:31.830787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:37:15.981551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066396+0000","last_change":"2024-03-11T21:54:52.433086+0000","last_active":"2024-03-11T21:55:32.066396+0000","last_peered":"2024-03-11T21:55:32.066396+0000","last_clean":"2024-03-11T21:55:32.066396+0000","last_became_active":"2024-03-11T21:54:52.432691+0000","last_became_peered":"2024-03-11T21:54:52.432691+0000","last_unstale":"2024-03-11T21:55:32.066396+0000","last_undegraded":"2024-03-11T21:55:32.066396+0000","last_fullsized":"2024-03-11T21:55:32.066396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:48:57.755182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831727+0000","last_change":"2024-03-11T21:54:56.451709+0000","last_active":"2024-03-11T21:55:31.831727+0000","last_peered":"2024-03-11T21:55:31.831727+0000","last_clean":"2024-03-11T21:55:31.831727+0000","last_became_active":"2024-03-11T21:54:56.451369+0000","last_became_peered":"2024-03-11T21:54:56.451369+0000","last_unstale":"2024-03-11T21:55:31.831727+0000","last_undegraded":"2024-03-11T21:55:31.831727+0000","last_fullsized":"2024-03-11T21:55:31.831727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:42.284323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066581+0000","last_change":"2024-03-11T21:54:54.449788+0000","last_active":"2024-03-11T21:55:32.066581+0000","last_peered":"2024-03-11T21:55:32.066581+0000","last_clean":"2024-03-11T21:55:32.066581+0000","last_became_active":"2024-03-11T21:54:54.449461+0000","last_became_peered":"2024-03-11T21:54:54.449461+0000","last_unstale":"2024-03-11T21:55:32.066581+0000","last_undegraded":"2024-03-11T21:55:32.066581+0000","last_fullsized":"2024-03-11T21:55:32.066581+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:55:53.595676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064816+0000","last_change":"2024-03-11T21:54:50.418827+0000","last_active":"2024-03-11T21:55:32.064816+0000","last_peered":"2024-03-11T21:55:32.064816+0000","last_clean":"2024-03-11T21:55:32.064816+0000","last_became_active":"2024-03-11T21:54:50.418535+0000","last_became_peered":"2024-03-11T21:54:50.418535+0000","last_unstale":"2024-03-11T21:55:32.064816+0000","last_undegraded":"2024-03-11T21:55:32.064816+0000","last_fullsized":"2024-03-11T21:55:32.064816+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:04:07.214347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.980544+0000","last_change":"2024-03-11T21:54:52.434361+0000","last_active":"2024-03-11T21:55:31.980544+0000","last_peered":"2024-03-11T21:55:31.980544+0000","last_clean":"2024-03-11T21:55:31.980544+0000","last_became_active":"2024-03-11T21:54:52.433631+0000","last_became_peered":"2024-03-11T21:54:52.433631+0000","last_unstale":"2024-03-11T21:55:31.980544+0000","last_undegraded":"2024-03-11T21:55:31.980544+0000","last_fullsized":"2024-03-11T21:55:31.980544+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:56:35.530845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946341+0000","last_change":"2024-03-11T21:54:56.456134+0000","last_active":"2024-03-11T21:55:31.946341+0000","last_peered":"2024-03-11T21:55:31.946341+0000","last_clean":"2024-03-11T21:55:31.946341+0000","last_became_active":"2024-03-11T21:54:56.455755+0000","last_became_peered":"2024-03-11T21:54:56.455755+0000","last_unstale":"2024-03-11T21:55:31.946341+0000","last_undegraded":"2024-03-11T21:55:31.946341+0000","last_fullsized":"2024-03-11T21:55:31.946341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:53:19.091447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.979902+0000","last_change":"2024-03-11T21:54:54.448627+0000","last_active":"2024-03-11T21:55:31.979902+0000","last_peered":"2024-03-11T21:55:31.979902+0000","last_clean":"2024-03-11T21:55:31.979902+0000","last_became_active":"2024-03-11T21:54:54.448314+0000","last_became_peered":"2024-03-11T21:54:54.448314+0000","last_unstale":"2024-03-11T21:55:31.979902+0000","last_undegraded":"2024-03-11T21:55:31.979902+0000","last_fullsized":"2024-03-11T21:55:31.979902+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:58:07.417041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065655+0000","last_change":"2024-03-11T21:54:50.421248+0000","last_active":"2024-03-11T21:55:32.065655+0000","last_peered":"2024-03-11T21:55:32.065655+0000","last_clean":"2024-03-11T21:55:32.065655+0000","last_became_active":"2024-03-11T21:54:50.420906+0000","last_became_peered":"2024-03-11T21:54:50.420906+0000","last_unstale":"2024-03-11T21:55:32.065655+0000","last_undegraded":"2024-03-11T21:55:32.065655+0000","last_fullsized":"2024-03-11T21:55:32.065655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:23.113143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064893+0000","last_change":"2024-03-11T21:54:52.437893+0000","last_active":"2024-03-11T21:55:32.064893+0000","last_peered":"2024-03-11T21:55:32.064893+0000","last_clean":"2024-03-11T21:55:32.064893+0000","last_became_active":"2024-03-11T21:54:52.437581+0000","last_became_peered":"2024-03-11T21:54:52.437581+0000","last_unstale":"2024-03-11T21:55:32.064893+0000","last_undegraded":"2024-03-11T21:55:32.064893+0000","last_fullsized":"2024-03-11T21:55:32.064893+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:04:39.231991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065520+0000","last_change":"2024-03-11T21:54:56.536583+0000","last_active":"2024-03-11T21:55:32.065520+0000","last_peered":"2024-03-11T21:55:32.065520+0000","last_clean":"2024-03-11T21:55:32.065520+0000","last_became_active":"2024-03-11T21:54:56.536202+0000","last_became_peered":"2024-03-11T21:54:56.536202+0000","last_unstale":"2024-03-11T21:55:32.065520+0000","last_undegraded":"2024-03-11T21:55:32.065520+0000","last_fullsized":"2024-03-11T21:55:32.065520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:49.413728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946524+0000","last_change":"2024-03-11T21:54:54.447995+0000","last_active":"2024-03-11T21:55:31.946524+0000","last_peered":"2024-03-11T21:55:31.946524+0000","last_clean":"2024-03-11T21:55:31.946524+0000","last_became_active":"2024-03-11T21:54:54.446803+0000","last_became_peered":"2024-03-11T21:54:54.446803+0000","last_unstale":"2024-03-11T21:55:31.946524+0000","last_undegraded":"2024-03-11T21:55:31.946524+0000","last_fullsized":"2024-03-11T21:55:31.946524+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:57:22.943370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934183+0000","last_change":"2024-03-11T21:54:50.412364+0000","last_active":"2024-03-11T21:55:31.934183+0000","last_peered":"2024-03-11T21:55:31.934183+0000","last_clean":"2024-03-11T21:55:31.934183+0000","last_became_active":"2024-03-11T21:54:50.411181+0000","last_became_peered":"2024-03-11T21:54:50.411181+0000","last_unstale":"2024-03-11T21:55:31.934183+0000","last_undegraded":"2024-03-11T21:55:31.934183+0000","last_fullsized":"2024-03-11T21:55:31.934183+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:07.041244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065672+0000","last_change":"2024-03-11T21:54:52.438828+0000","last_active":"2024-03-11T21:55:32.065672+0000","last_peered":"2024-03-11T21:55:32.065672+0000","last_clean":"2024-03-11T21:55:32.065672+0000","last_became_active":"2024-03-11T21:54:52.438187+0000","last_became_peered":"2024-03-11T21:54:52.438187+0000","last_unstale":"2024-03-11T21:55:32.065672+0000","last_undegraded":"2024-03-11T21:55:32.065672+0000","last_fullsized":"2024-03-11T21:55:32.065672+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:12:09.221931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064854+0000","last_change":"2024-03-11T21:54:56.458260+0000","last_active":"2024-03-11T21:55:32.064854+0000","last_peered":"2024-03-11T21:55:32.064854+0000","last_clean":"2024-03-11T21:55:32.064854+0000","last_became_active":"2024-03-11T21:54:56.457458+0000","last_became_peered":"2024-03-11T21:54:56.457458+0000","last_unstale":"2024-03-11T21:55:32.064854+0000","last_undegraded":"2024-03-11T21:55:32.064854+0000","last_fullsized":"2024-03-11T21:55:32.064854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:22:19.422414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065704+0000","last_change":"2024-03-11T21:54:54.447062+0000","last_active":"2024-03-11T21:55:32.065704+0000","last_peered":"2024-03-11T21:55:32.065704+0000","last_clean":"2024-03-11T21:55:32.065704+0000","last_became_active":"2024-03-11T21:54:54.446750+0000","last_became_peered":"2024-03-11T21:54:54.446750+0000","last_unstale":"2024-03-11T21:55:32.065704+0000","last_undegraded":"2024-03-11T21:55:32.065704+0000","last_fullsized":"2024-03-11T21:55:32.065704+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:17:28.010411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946671+0000","last_change":"2024-03-11T21:54:50.420855+0000","last_active":"2024-03-11T21:55:31.946671+0000","last_peered":"2024-03-11T21:55:31.946671+0000","last_clean":"2024-03-11T21:55:31.946671+0000","last_became_active":"2024-03-11T21:54:50.420404+0000","last_became_peered":"2024-03-11T21:54:50.420404+0000","last_unstale":"2024-03-11T21:55:31.946671+0000","last_undegraded":"2024-03-11T21:55:31.946671+0000","last_fullsized":"2024-03-11T21:55:31.946671+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:25:28.405026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831247+0000","last_change":"2024-03-11T21:54:52.432141+0000","last_active":"2024-03-11T21:55:31.831247+0000","last_peered":"2024-03-11T21:55:31.831247+0000","last_clean":"2024-03-11T21:55:31.831247+0000","last_became_active":"2024-03-11T21:54:52.431667+0000","last_became_peered":"2024-03-11T21:54:52.431667+0000","last_unstale":"2024-03-11T21:55:31.831247+0000","last_undegraded":"2024-03-11T21:55:31.831247+0000","last_fullsized":"2024-03-11T21:55:31.831247+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:39.141139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064911+0000","last_change":"2024-03-11T21:54:56.462182+0000","last_active":"2024-03-11T21:55:32.064911+0000","last_peered":"2024-03-11T21:55:32.064911+0000","last_clean":"2024-03-11T21:55:32.064911+0000","last_became_active":"2024-03-11T21:54:56.460594+0000","last_became_peered":"2024-03-11T21:54:56.460594+0000","last_unstale":"2024-03-11T21:55:32.064911+0000","last_undegraded":"2024-03-11T21:55:32.064911+0000","last_fullsized":"2024-03-11T21:55:32.064911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:26.691208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831035+0000","last_change":"2024-03-11T21:54:54.447234+0000","last_active":"2024-03-11T21:55:31.831035+0000","last_peered":"2024-03-11T21:55:31.831035+0000","last_clean":"2024-03-11T21:55:31.831035+0000","last_became_active":"2024-03-11T21:54:54.446915+0000","last_became_peered":"2024-03-11T21:54:54.446915+0000","last_unstale":"2024-03-11T21:55:31.831035+0000","last_undegraded":"2024-03-11T21:55:31.831035+0000","last_fullsized":"2024-03-11T21:55:31.831035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:14.615758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831765+0000","last_change":"2024-03-11T21:54:50.415621+0000","last_active":"2024-03-11T21:55:31.831765+0000","last_peered":"2024-03-11T21:55:31.831765+0000","last_clean":"2024-03-11T21:55:31.831765+0000","last_became_active":"2024-03-11T21:54:50.414331+0000","last_became_peered":"2024-03-11T21:54:50.414331+0000","last_unstale":"2024-03-11T21:55:31.831765+0000","last_undegraded":"2024-03-11T21:55:31.831765+0000","last_fullsized":"2024-03-11T21:55:31.831765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:34:44.860393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831375+0000","last_change":"2024-03-11T21:54:52.434309+0000","last_active":"2024-03-11T21:55:31.831375+0000","last_peered":"2024-03-11T21:55:31.831375+0000","last_clean":"2024-03-11T21:55:31.831375+0000","last_became_active":"2024-03-11T21:54:52.434033+0000","last_became_peered":"2024-03-11T21:54:52.434033+0000","last_unstale":"2024-03-11T21:55:31.831375+0000","last_undegraded":"2024-03-11T21:55:31.831375+0000","last_fullsized":"2024-03-11T21:55:31.831375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:19.155542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066094+0000","last_change":"2024-03-11T21:54:56.462310+0000","last_active":"2024-03-11T21:55:32.066094+0000","last_peered":"2024-03-11T21:55:32.066094+0000","last_clean":"2024-03-11T21:55:32.066094+0000","last_became_active":"2024-03-11T21:54:56.461998+0000","last_became_peered":"2024-03-11T21:54:56.461998+0000","last_unstale":"2024-03-11T21:55:32.066094+0000","last_undegraded":"2024-03-11T21:55:32.066094+0000","last_fullsized":"2024-03-11T21:55:32.066094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:51.931704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472185+0000","last_change":"2024-03-11T21:54:54.450939+0000","last_active":"2024-03-11T21:56:05.472185+0000","last_peered":"2024-03-11T21:56:05.472185+0000","last_clean":"2024-03-11T21:56:05.472185+0000","last_became_active":"2024-03-11T21:54:54.450608+0000","last_became_peered":"2024-03-11T21:54:54.450608+0000","last_unstale":"2024-03-11T21:56:05.472185+0000","last_undegraded":"2024-03-11T21:56:05.472185+0000","last_fullsized":"2024-03-11T21:56:05.472185+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:58:39.219918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946500+0000","last_change":"2024-03-11T21:54:50.414322+0000","last_active":"2024-03-11T21:55:31.946500+0000","last_peered":"2024-03-11T21:55:31.946500+0000","last_clean":"2024-03-11T21:55:31.946500+0000","last_became_active":"2024-03-11T21:54:50.413934+0000","last_became_peered":"2024-03-11T21:54:50.413934+0000","last_unstale":"2024-03-11T21:55:31.946500+0000","last_undegraded":"2024-03-11T21:55:31.946500+0000","last_fullsized":"2024-03-11T21:55:31.946500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:51.320184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065957+0000","last_change":"2024-03-11T21:54:52.436118+0000","last_active":"2024-03-11T21:55:32.065957+0000","last_peered":"2024-03-11T21:55:32.065957+0000","last_clean":"2024-03-11T21:55:32.065957+0000","last_became_active":"2024-03-11T21:54:52.434857+0000","last_became_peered":"2024-03-11T21:54:52.434857+0000","last_unstale":"2024-03-11T21:55:32.065957+0000","last_undegraded":"2024-03-11T21:55:32.065957+0000","last_fullsized":"2024-03-11T21:55:32.065957+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:46:52.183401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831706+0000","last_change":"2024-03-11T21:54:56.450231+0000","last_active":"2024-03-11T21:55:31.831706+0000","last_peered":"2024-03-11T21:55:31.831706+0000","last_clean":"2024-03-11T21:55:31.831706+0000","last_became_active":"2024-03-11T21:54:56.449467+0000","last_became_peered":"2024-03-11T21:54:56.449467+0000","last_unstale":"2024-03-11T21:55:31.831706+0000","last_undegraded":"2024-03-11T21:55:31.831706+0000","last_fullsized":"2024-03-11T21:55:31.831706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:06.062812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472005+0000","last_change":"2024-03-11T21:54:54.452707+0000","last_active":"2024-03-11T21:56:05.472005+0000","last_peered":"2024-03-11T21:56:05.472005+0000","last_clean":"2024-03-11T21:56:05.472005+0000","last_became_active":"2024-03-11T21:54:54.452185+0000","last_became_peered":"2024-03-11T21:54:54.452185+0000","last_unstale":"2024-03-11T21:56:05.472005+0000","last_undegraded":"2024-03-11T21:56:05.472005+0000","last_fullsized":"2024-03-11T21:56:05.472005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:02:49.662461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065259+0000","last_change":"2024-03-11T21:54:50.421489+0000","last_active":"2024-03-11T21:55:32.065259+0000","last_peered":"2024-03-11T21:55:32.065259+0000","last_clean":"2024-03-11T21:55:32.065259+0000","last_became_active":"2024-03-11T21:54:50.420942+0000","last_became_peered":"2024-03-11T21:54:50.420942+0000","last_unstale":"2024-03-11T21:55:32.065259+0000","last_undegraded":"2024-03-11T21:55:32.065259+0000","last_fullsized":"2024-03-11T21:55:32.065259+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:50:36.980863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934369+0000","last_change":"2024-03-11T21:54:52.433167+0000","last_active":"2024-03-11T21:55:31.934369+0000","last_peered":"2024-03-11T21:55:31.934369+0000","last_clean":"2024-03-11T21:55:31.934369+0000","last_became_active":"2024-03-11T21:54:52.432837+0000","last_became_peered":"2024-03-11T21:54:52.432837+0000","last_unstale":"2024-03-11T21:55:31.934369+0000","last_undegraded":"2024-03-11T21:55:31.934369+0000","last_fullsized":"2024-03-11T21:55:31.934369+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:35:05.309569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832789+0000","last_change":"2024-03-11T21:54:56.453995+0000","last_active":"2024-03-11T21:55:31.832789+0000","last_peered":"2024-03-11T21:55:31.832789+0000","last_clean":"2024-03-11T21:55:31.832789+0000","last_became_active":"2024-03-11T21:54:56.453694+0000","last_became_peered":"2024-03-11T21:54:56.453694+0000","last_unstale":"2024-03-11T21:55:31.832789+0000","last_undegraded":"2024-03-11T21:55:31.832789+0000","last_fullsized":"2024-03-11T21:55:31.832789+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:42.432711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066825+0000","last_change":"2024-03-11T21:54:54.450613+0000","last_active":"2024-03-11T21:55:32.066825+0000","last_peered":"2024-03-11T21:55:32.066825+0000","last_clean":"2024-03-11T21:55:32.066825+0000","last_became_active":"2024-03-11T21:54:54.450311+0000","last_became_peered":"2024-03-11T21:54:54.450311+0000","last_unstale":"2024-03-11T21:55:32.066825+0000","last_undegraded":"2024-03-11T21:55:32.066825+0000","last_fullsized":"2024-03-11T21:55:32.066825+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:37:12.252483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946353+0000","last_change":"2024-03-11T21:54:50.414818+0000","last_active":"2024-03-11T21:55:31.946353+0000","last_peered":"2024-03-11T21:55:31.946353+0000","last_clean":"2024-03-11T21:55:31.946353+0000","last_became_active":"2024-03-11T21:54:50.414529+0000","last_became_peered":"2024-03-11T21:54:50.414529+0000","last_unstale":"2024-03-11T21:55:31.946353+0000","last_undegraded":"2024-03-11T21:55:31.946353+0000","last_fullsized":"2024-03-11T21:55:31.946353+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:22:43.069361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066450+0000","last_change":"2024-03-11T21:54:52.433312+0000","last_active":"2024-03-11T21:55:32.066450+0000","last_peered":"2024-03-11T21:55:32.066450+0000","last_clean":"2024-03-11T21:55:32.066450+0000","last_became_active":"2024-03-11T21:54:52.432982+0000","last_became_peered":"2024-03-11T21:54:52.432982+0000","last_unstale":"2024-03-11T21:55:32.066450+0000","last_undegraded":"2024-03-11T21:55:32.066450+0000","last_fullsized":"2024-03-11T21:55:32.066450+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:11:02.802808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946716+0000","last_change":"2024-03-11T21:54:56.538240+0000","last_active":"2024-03-11T21:55:31.946716+0000","last_peered":"2024-03-11T21:55:31.946716+0000","last_clean":"2024-03-11T21:55:31.946716+0000","last_became_active":"2024-03-11T21:54:56.537946+0000","last_became_peered":"2024-03-11T21:54:56.537946+0000","last_unstale":"2024-03-11T21:55:31.946716+0000","last_undegraded":"2024-03-11T21:55:31.946716+0000","last_fullsized":"2024-03-11T21:55:31.946716+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:08:37.097383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934253+0000","last_change":"2024-03-11T21:54:54.453321+0000","last_active":"2024-03-11T21:55:31.934253+0000","last_peered":"2024-03-11T21:55:31.934253+0000","last_clean":"2024-03-11T21:55:31.934253+0000","last_became_active":"2024-03-11T21:54:54.453006+0000","last_became_peered":"2024-03-11T21:54:54.453006+0000","last_unstale":"2024-03-11T21:55:31.934253+0000","last_undegraded":"2024-03-11T21:55:31.934253+0000","last_fullsized":"2024-03-11T21:55:31.934253+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:03:18.399494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.979847+0000","last_change":"2024-03-11T21:54:50.418065+0000","last_active":"2024-03-11T21:55:31.979847+0000","last_peered":"2024-03-11T21:55:31.979847+0000","last_clean":"2024-03-11T21:55:31.979847+0000","last_became_active":"2024-03-11T21:54:50.417634+0000","last_became_peered":"2024-03-11T21:54:50.417634+0000","last_unstale":"2024-03-11T21:55:31.979847+0000","last_undegraded":"2024-03-11T21:55:31.979847+0000","last_fullsized":"2024-03-11T21:55:31.979847+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:28.038598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":38,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066855+0000","last_change":"2024-03-11T21:54:52.434550+0000","last_active":"2024-03-11T21:55:32.066855+0000","last_peered":"2024-03-11T21:55:32.066855+0000","last_clean":"2024-03-11T21:55:32.066855+0000","last_became_active":"2024-03-11T21:54:52.434240+0000","last_became_peered":"2024-03-11T21:54:52.434240+0000","last_unstale":"2024-03-11T21:55:32.066855+0000","last_undegraded":"2024-03-11T21:55:32.066855+0000","last_fullsized":"2024-03-11T21:55:32.066855+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:38:08.562054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933959+0000","last_change":"2024-03-11T21:54:56.535781+0000","last_active":"2024-03-11T21:55:31.933959+0000","last_peered":"2024-03-11T21:55:31.933959+0000","last_clean":"2024-03-11T21:55:31.933959+0000","last_became_active":"2024-03-11T21:54:56.535484+0000","last_became_peered":"2024-03-11T21:54:56.535484+0000","last_unstale":"2024-03-11T21:55:31.933959+0000","last_undegraded":"2024-03-11T21:55:31.933959+0000","last_fullsized":"2024-03-11T21:55:31.933959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:23.390563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.947125+0000","last_change":"2024-03-11T21:54:50.420247+0000","last_active":"2024-03-11T21:55:31.947125+0000","last_peered":"2024-03-11T21:55:31.947125+0000","last_clean":"2024-03-11T21:55:31.947125+0000","last_became_active":"2024-03-11T21:54:50.419950+0000","last_became_peered":"2024-03-11T21:54:50.419950+0000","last_unstale":"2024-03-11T21:55:31.947125+0000","last_undegraded":"2024-03-11T21:55:31.947125+0000","last_fullsized":"2024-03-11T21:55:31.947125+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:59:27.644018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933690+0000","last_change":"2024-03-11T21:54:52.430702+0000","last_active":"2024-03-11T21:55:31.933690+0000","last_peered":"2024-03-11T21:55:31.933690+0000","last_clean":"2024-03-11T21:55:31.933690+0000","last_became_active":"2024-03-11T21:54:52.430384+0000","last_became_peered":"2024-03-11T21:54:52.430384+0000","last_unstale":"2024-03-11T21:55:31.933690+0000","last_undegraded":"2024-03-11T21:55:31.933690+0000","last_fullsized":"2024-03-11T21:55:31.933690+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:44:08.784012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472151+0000","last_change":"2024-03-11T21:54:54.455583+0000","last_active":"2024-03-11T21:56:05.472151+0000","last_peered":"2024-03-11T21:56:05.472151+0000","last_clean":"2024-03-11T21:56:05.472151+0000","last_became_active":"2024-03-11T21:54:54.455284+0000","last_became_peered":"2024-03-11T21:54:54.455284+0000","last_unstale":"2024-03-11T21:56:05.472151+0000","last_undegraded":"2024-03-11T21:56:05.472151+0000","last_fullsized":"2024-03-11T21:56:05.472151+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:38:26.161240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065609+0000","last_change":"2024-03-11T21:54:56.461870+0000","last_active":"2024-03-11T21:55:32.065609+0000","last_peered":"2024-03-11T21:55:32.065609+0000","last_clean":"2024-03-11T21:55:32.065609+0000","last_became_active":"2024-03-11T21:54:56.460346+0000","last_became_peered":"2024-03-11T21:54:56.460346+0000","last_unstale":"2024-03-11T21:55:32.065609+0000","last_undegraded":"2024-03-11T21:55:32.065609+0000","last_fullsized":"2024-03-11T21:55:32.065609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:06:43.135842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.947030+0000","last_change":"2024-03-11T21:54:50.415553+0000","last_active":"2024-03-11T21:55:31.947030+0000","last_peered":"2024-03-11T21:55:31.947030+0000","last_clean":"2024-03-11T21:55:31.947030+0000","last_became_active":"2024-03-11T21:54:50.414908+0000","last_became_peered":"2024-03-11T21:54:50.414908+0000","last_unstale":"2024-03-11T21:55:31.947030+0000","last_undegraded":"2024-03-11T21:55:31.947030+0000","last_fullsized":"2024-03-11T21:55:31.947030+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:54.526575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831808+0000","last_change":"2024-03-11T21:54:52.437879+0000","last_active":"2024-03-11T21:55:31.831808+0000","last_peered":"2024-03-11T21:55:31.831808+0000","last_clean":"2024-03-11T21:55:31.831808+0000","last_became_active":"2024-03-11T21:54:52.437569+0000","last_became_peered":"2024-03-11T21:54:52.437569+0000","last_unstale":"2024-03-11T21:55:31.831808+0000","last_undegraded":"2024-03-11T21:55:31.831808+0000","last_fullsized":"2024-03-11T21:55:31.831808+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:31.980225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832232+0000","last_change":"2024-03-11T21:54:54.448595+0000","last_active":"2024-03-11T21:55:31.832232+0000","last_peered":"2024-03-11T21:55:31.832232+0000","last_clean":"2024-03-11T21:55:31.832232+0000","last_became_active":"2024-03-11T21:54:54.443571+0000","last_became_peered":"2024-03-11T21:54:54.443571+0000","last_unstale":"2024-03-11T21:55:31.832232+0000","last_undegraded":"2024-03-11T21:55:31.832232+0000","last_fullsized":"2024-03-11T21:55:31.832232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:44.205274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946214+0000","last_change":"2024-03-11T21:54:56.460951+0000","last_active":"2024-03-11T21:55:31.946214+0000","last_peered":"2024-03-11T21:55:31.946214+0000","last_clean":"2024-03-11T21:55:31.946214+0000","last_became_active":"2024-03-11T21:54:56.460663+0000","last_became_peered":"2024-03-11T21:54:56.460663+0000","last_unstale":"2024-03-11T21:55:31.946214+0000","last_undegraded":"2024-03-11T21:55:31.946214+0000","last_fullsized":"2024-03-11T21:55:31.946214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:05:28.165394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832582+0000","last_change":"2024-03-11T21:54:54.448661+0000","last_active":"2024-03-11T21:55:31.832582+0000","last_peered":"2024-03-11T21:55:31.832582+0000","last_clean":"2024-03-11T21:55:31.832582+0000","last_became_active":"2024-03-11T21:54:54.448220+0000","last_became_peered":"2024-03-11T21:54:54.448220+0000","last_unstale":"2024-03-11T21:55:31.832582+0000","last_undegraded":"2024-03-11T21:55:31.832582+0000","last_fullsized":"2024-03-11T21:55:31.832582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:14:20.312541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946873+0000","last_change":"2024-03-11T21:54:50.418687+0000","last_active":"2024-03-11T21:55:31.946873+0000","last_peered":"2024-03-11T21:55:31.946873+0000","last_clean":"2024-03-11T21:55:31.946873+0000","last_became_active":"2024-03-11T21:54:50.418312+0000","last_became_peered":"2024-03-11T21:54:50.418312+0000","last_unstale":"2024-03-11T21:55:31.946873+0000","last_undegraded":"2024-03-11T21:55:31.946873+0000","last_fullsized":"2024-03-11T21:55:31.946873+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:45.376240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.980194+0000","last_change":"2024-03-11T21:54:52.430399+0000","last_active":"2024-03-11T21:55:31.980194+0000","last_peered":"2024-03-11T21:55:31.980194+0000","last_clean":"2024-03-11T21:55:31.980194+0000","last_became_active":"2024-03-11T21:54:52.430095+0000","last_became_peered":"2024-03-11T21:54:52.430095+0000","last_unstale":"2024-03-11T21:55:31.980194+0000","last_undegraded":"2024-03-11T21:55:31.980194+0000","last_fullsized":"2024-03-11T21:55:31.980194+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:12:40.577667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065124+0000","last_change":"2024-03-11T21:54:56.457418+0000","last_active":"2024-03-11T21:55:32.065124+0000","last_peered":"2024-03-11T21:55:32.065124+0000","last_clean":"2024-03-11T21:55:32.065124+0000","last_became_active":"2024-03-11T21:54:56.457005+0000","last_became_peered":"2024-03-11T21:54:56.457005+0000","last_unstale":"2024-03-11T21:55:32.065124+0000","last_undegraded":"2024-03-11T21:55:32.065124+0000","last_fullsized":"2024-03-11T21:55:32.065124+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:15:02.771265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832447+0000","last_change":"2024-03-11T21:54:54.446607+0000","last_active":"2024-03-11T21:55:31.832447+0000","last_peered":"2024-03-11T21:55:31.832447+0000","last_clean":"2024-03-11T21:55:31.832447+0000","last_became_active":"2024-03-11T21:54:54.446273+0000","last_became_peered":"2024-03-11T21:54:54.446273+0000","last_unstale":"2024-03-11T21:55:31.832447+0000","last_undegraded":"2024-03-11T21:55:31.832447+0000","last_fullsized":"2024-03-11T21:55:31.832447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:34:35.240809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934289+0000","last_change":"2024-03-11T21:54:50.416492+0000","last_active":"2024-03-11T21:55:31.934289+0000","last_peered":"2024-03-11T21:55:31.934289+0000","last_clean":"2024-03-11T21:55:31.934289+0000","last_became_active":"2024-03-11T21:54:50.416197+0000","last_became_peered":"2024-03-11T21:54:50.416197+0000","last_unstale":"2024-03-11T21:55:31.934289+0000","last_undegraded":"2024-03-11T21:55:31.934289+0000","last_fullsized":"2024-03-11T21:55:31.934289+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:07:12.077530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066666+0000","last_change":"2024-03-11T21:54:52.431538+0000","last_active":"2024-03-11T21:55:32.066666+0000","last_peered":"2024-03-11T21:55:32.066666+0000","last_clean":"2024-03-11T21:55:32.066666+0000","last_became_active":"2024-03-11T21:54:52.431141+0000","last_became_peered":"2024-03-11T21:54:52.431141+0000","last_unstale":"2024-03-11T21:55:32.066666+0000","last_undegraded":"2024-03-11T21:55:32.066666+0000","last_fullsized":"2024-03-11T21:55:32.066666+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:32:12.661431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066689+0000","last_change":"2024-03-11T21:54:56.460842+0000","last_active":"2024-03-11T21:55:32.066689+0000","last_peered":"2024-03-11T21:55:32.066689+0000","last_clean":"2024-03-11T21:55:32.066689+0000","last_became_active":"2024-03-11T21:54:56.460512+0000","last_became_peered":"2024-03-11T21:54:56.460512+0000","last_unstale":"2024-03-11T21:55:32.066689+0000","last_undegraded":"2024-03-11T21:55:32.066689+0000","last_fullsized":"2024-03-11T21:55:32.066689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:28:48.717744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933800+0000","last_change":"2024-03-11T21:54:54.443087+0000","last_active":"2024-03-11T21:55:31.933800+0000","last_peered":"2024-03-11T21:55:31.933800+0000","last_clean":"2024-03-11T21:55:31.933800+0000","last_became_active":"2024-03-11T21:54:50.413769+0000","last_became_peered":"2024-03-11T21:54:50.413769+0000","last_unstale":"2024-03-11T21:55:31.933800+0000","last_undegraded":"2024-03-11T21:55:31.933800+0000","last_fullsized":"2024-03-11T21:55:31.933800+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:18.090345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020997749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831229+0000","last_change":"2024-03-11T21:54:50.417454+0000","last_active":"2024-03-11T21:55:31.831229+0000","last_peered":"2024-03-11T21:55:31.831229+0000","last_clean":"2024-03-11T21:55:31.831229+0000","last_became_active":"2024-03-11T21:54:50.417085+0000","last_became_peered":"2024-03-11T21:54:50.417085+0000","last_unstale":"2024-03-11T21:55:31.831229+0000","last_undegraded":"2024-03-11T21:55:31.831229+0000","last_fullsized":"2024-03-11T21:55:31.831229+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:20:17.362116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"71'29","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472152+0000","last_change":"2024-03-11T21:55:13.523981+0000","last_active":"2024-03-11T21:56:05.472152+0000","last_peered":"2024-03-11T21:56:05.472152+0000","last_clean":"2024-03-11T21:56:05.472152+0000","last_became_active":"2024-03-11T21:55:13.509349+0000","last_became_peered":"2024-03-11T21:55:13.509349+0000","last_unstale":"2024-03-11T21:56:05.472152+0000","last_undegraded":"2024-03-11T21:56:05.472152+0000","last_fullsized":"2024-03-11T21:56:05.472152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:12:14.846827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065665+0000","last_change":"2024-03-11T21:54:54.449510+0000","last_active":"2024-03-11T21:55:32.065665+0000","last_peered":"2024-03-11T21:55:32.065665+0000","last_clean":"2024-03-11T21:55:32.065665+0000","last_became_active":"2024-03-11T21:54:54.449098+0000","last_became_peered":"2024-03-11T21:54:54.449098+0000","last_unstale":"2024-03-11T21:55:32.065665+0000","last_undegraded":"2024-03-11T21:55:32.065665+0000","last_fullsized":"2024-03-11T21:55:32.065665+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:06:38.828874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946961+0000","last_change":"2024-03-11T21:54:56.535374+0000","last_active":"2024-03-11T21:55:31.946961+0000","last_peered":"2024-03-11T21:55:31.946961+0000","last_clean":"2024-03-11T21:55:31.946961+0000","last_became_active":"2024-03-11T21:54:56.535062+0000","last_became_peered":"2024-03-11T21:54:56.535062+0000","last_unstale":"2024-03-11T21:55:31.946961+0000","last_undegraded":"2024-03-11T21:55:31.946961+0000","last_fullsized":"2024-03-11T21:55:31.946961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:52:35.444035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830967+0000","last_change":"2024-03-11T21:54:50.415222+0000","last_active":"2024-03-11T21:55:31.830967+0000","last_peered":"2024-03-11T21:55:31.830967+0000","last_clean":"2024-03-11T21:55:31.830967+0000","last_became_active":"2024-03-11T21:54:50.414931+0000","last_became_peered":"2024-03-11T21:54:50.414931+0000","last_unstale":"2024-03-11T21:55:31.830967+0000","last_undegraded":"2024-03-11T21:55:31.830967+0000","last_fullsized":"2024-03-11T21:55:31.830967+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:43:28.948730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832471+0000","last_change":"2024-03-11T21:54:54.436020+0000","last_active":"2024-03-11T21:55:31.832471+0000","last_peered":"2024-03-11T21:55:31.832471+0000","last_clean":"2024-03-11T21:55:31.832471+0000","last_became_active":"2024-03-11T21:54:50.415862+0000","last_became_peered":"2024-03-11T21:54:50.415862+0000","last_unstale":"2024-03-11T21:55:31.832471+0000","last_undegraded":"2024-03-11T21:55:31.832471+0000","last_fullsized":"2024-03-11T21:55:31.832471+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:42:38.179547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013831500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832995+0000","last_change":"2024-03-11T21:55:13.516801+0000","last_active":"2024-03-11T21:55:31.832995+0000","last_peered":"2024-03-11T21:55:31.832995+0000","last_clean":"2024-03-11T21:55:31.832995+0000","last_became_active":"2024-03-11T21:55:13.508331+0000","last_became_peered":"2024-03-11T21:55:13.508331+0000","last_unstale":"2024-03-11T21:55:31.832995+0000","last_undegraded":"2024-03-11T21:55:31.832995+0000","last_fullsized":"2024-03-11T21:55:31.832995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:00.214523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832774+0000","last_change":"2024-03-11T21:54:54.449009+0000","last_active":"2024-03-11T21:55:31.832774+0000","last_peered":"2024-03-11T21:55:31.832774+0000","last_clean":"2024-03-11T21:55:31.832774+0000","last_became_active":"2024-03-11T21:54:54.448646+0000","last_became_peered":"2024-03-11T21:54:54.448646+0000","last_unstale":"2024-03-11T21:55:31.832774+0000","last_undegraded":"2024-03-11T21:55:31.832774+0000","last_fullsized":"2024-03-11T21:55:31.832774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:21:16.719560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934325+0000","last_change":"2024-03-11T21:54:56.461733+0000","last_active":"2024-03-11T21:55:31.934325+0000","last_peered":"2024-03-11T21:55:31.934325+0000","last_clean":"2024-03-11T21:55:31.934325+0000","last_became_active":"2024-03-11T21:54:56.461429+0000","last_became_peered":"2024-03-11T21:54:56.461429+0000","last_unstale":"2024-03-11T21:55:31.934325+0000","last_undegraded":"2024-03-11T21:55:31.934325+0000","last_fullsized":"2024-03-11T21:55:31.934325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:25.346326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'188","reported_seq":345,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:33.928751+0000","last_change":"2024-03-11T21:54:40.647817+0000","last_active":"2024-03-11T21:55:33.928751+0000","last_peered":"2024-03-11T21:55:33.928751+0000","last_clean":"2024-03-11T21:55:33.928751+0000","last_became_active":"2024-03-11T21:54:40.647391+0000","last_became_peered":"2024-03-11T21:54:40.647391+0000","last_unstale":"2024-03-11T21:55:33.928751+0000","last_undegraded":"2024-03-11T21:55:33.928751+0000","last_fullsized":"2024-03-11T21:55:33.928751+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:52:26.665210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:52:26.665210+0000","last_clean_scrub_stamp":"2024-03-11T21:52:26.665210+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:43:36.382773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065564+0000","last_change":"2024-03-11T21:54:50.421030+0000","last_active":"2024-03-11T21:55:32.065564+0000","last_peered":"2024-03-11T21:55:32.065564+0000","last_clean":"2024-03-11T21:55:32.065564+0000","last_became_active":"2024-03-11T21:54:50.420706+0000","last_became_peered":"2024-03-11T21:54:50.420706+0000","last_unstale":"2024-03-11T21:55:32.065564+0000","last_undegraded":"2024-03-11T21:55:32.065564+0000","last_fullsized":"2024-03-11T21:55:32.065564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:57:25.226971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472184+0000","last_change":"2024-03-11T21:54:52.433823+0000","last_active":"2024-03-11T21:56:05.472184+0000","last_peered":"2024-03-11T21:56:05.472184+0000","last_clean":"2024-03-11T21:56:05.472184+0000","last_became_active":"2024-03-11T21:54:52.432703+0000","last_became_peered":"2024-03-11T21:54:52.432703+0000","last_unstale":"2024-03-11T21:56:05.472184+0000","last_undegraded":"2024-03-11T21:56:05.472184+0000","last_fullsized":"2024-03-11T21:56:05.472184+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:02.400690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946696+0000","last_change":"2024-03-11T21:54:54.457179+0000","last_active":"2024-03-11T21:55:31.946696+0000","last_peered":"2024-03-11T21:55:31.946696+0000","last_clean":"2024-03-11T21:55:31.946696+0000","last_became_active":"2024-03-11T21:54:54.456871+0000","last_became_peered":"2024-03-11T21:54:54.456871+0000","last_unstale":"2024-03-11T21:55:31.946696+0000","last_undegraded":"2024-03-11T21:55:31.946696+0000","last_fullsized":"2024-03-11T21:55:31.946696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:37.721150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831238+0000","last_change":"2024-03-11T21:54:56.461667+0000","last_active":"2024-03-11T21:55:31.831238+0000","last_peered":"2024-03-11T21:55:31.831238+0000","last_clean":"2024-03-11T21:55:31.831238+0000","last_became_active":"2024-03-11T21:54:56.461406+0000","last_became_peered":"2024-03-11T21:54:56.461406+0000","last_unstale":"2024-03-11T21:55:31.831238+0000","last_undegraded":"2024-03-11T21:55:31.831238+0000","last_fullsized":"2024-03-11T21:55:31.831238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:36:21.434251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065457+0000","last_change":"2024-03-11T21:54:50.419579+0000","last_active":"2024-03-11T21:55:32.065457+0000","last_peered":"2024-03-11T21:55:32.065457+0000","last_clean":"2024-03-11T21:55:32.065457+0000","last_became_active":"2024-03-11T21:54:50.418728+0000","last_became_peered":"2024-03-11T21:54:50.418728+0000","last_unstale":"2024-03-11T21:55:32.065457+0000","last_undegraded":"2024-03-11T21:55:32.065457+0000","last_fullsized":"2024-03-11T21:55:32.065457+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:37:43.186317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":119,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:11.506549+0000","last_change":"2024-03-11T21:54:54.443746+0000","last_active":"2024-03-11T21:56:11.506549+0000","last_peered":"2024-03-11T21:56:11.506549+0000","last_clean":"2024-03-11T21:56:11.506549+0000","last_became_active":"2024-03-11T21:54:50.416729+0000","last_became_peered":"2024-03-11T21:54:50.416729+0000","last_unstale":"2024-03-11T21:56:11.506549+0000","last_undegraded":"2024-03-11T21:56:11.506549+0000","last_fullsized":"2024-03-11T21:56:11.506549+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:55:33.852709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043829300000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066033+0000","last_change":"2024-03-11T21:54:52.434522+0000","last_active":"2024-03-11T21:55:32.066033+0000","last_peered":"2024-03-11T21:55:32.066033+0000","last_clean":"2024-03-11T21:55:32.066033+0000","last_became_active":"2024-03-11T21:54:52.434192+0000","last_became_peered":"2024-03-11T21:54:52.434192+0000","last_unstale":"2024-03-11T21:55:32.066033+0000","last_undegraded":"2024-03-11T21:55:32.066033+0000","last_fullsized":"2024-03-11T21:55:32.066033+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:50.551618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934531+0000","last_change":"2024-03-11T21:54:54.450785+0000","last_active":"2024-03-11T21:55:31.934531+0000","last_peered":"2024-03-11T21:55:31.934531+0000","last_clean":"2024-03-11T21:55:31.934531+0000","last_became_active":"2024-03-11T21:54:54.450478+0000","last_became_peered":"2024-03-11T21:54:54.450478+0000","last_unstale":"2024-03-11T21:55:31.934531+0000","last_undegraded":"2024-03-11T21:55:31.934531+0000","last_fullsized":"2024-03-11T21:55:31.934531+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:45:17.941514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830654+0000","last_change":"2024-03-11T21:54:56.535560+0000","last_active":"2024-03-11T21:55:31.830654+0000","last_peered":"2024-03-11T21:55:31.830654+0000","last_clean":"2024-03-11T21:55:31.830654+0000","last_became_active":"2024-03-11T21:54:56.535161+0000","last_became_peered":"2024-03-11T21:54:56.535161+0000","last_unstale":"2024-03-11T21:55:31.830654+0000","last_undegraded":"2024-03-11T21:55:31.830654+0000","last_fullsized":"2024-03-11T21:55:31.830654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:00:27.195839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831383+0000","last_change":"2024-03-11T21:54:50.413755+0000","last_active":"2024-03-11T21:55:31.831383+0000","last_peered":"2024-03-11T21:55:31.831383+0000","last_clean":"2024-03-11T21:55:31.831383+0000","last_became_active":"2024-03-11T21:54:50.413453+0000","last_became_peered":"2024-03-11T21:54:50.413453+0000","last_unstale":"2024-03-11T21:55:31.831383+0000","last_undegraded":"2024-03-11T21:55:31.831383+0000","last_fullsized":"2024-03-11T21:55:31.831383+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:31:41.724551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":65,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066411+0000","last_change":"2024-03-11T21:54:52.430361+0000","last_active":"2024-03-11T21:55:32.066411+0000","last_peered":"2024-03-11T21:55:32.066411+0000","last_clean":"2024-03-11T21:55:32.066411+0000","last_became_active":"2024-03-11T21:54:52.429990+0000","last_became_peered":"2024-03-11T21:54:52.429990+0000","last_unstale":"2024-03-11T21:55:32.066411+0000","last_undegraded":"2024-03-11T21:55:32.066411+0000","last_fullsized":"2024-03-11T21:55:32.066411+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:56:30.808133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.980248+0000","last_change":"2024-03-11T21:54:54.451141+0000","last_active":"2024-03-11T21:55:31.980248+0000","last_peered":"2024-03-11T21:55:31.980248+0000","last_clean":"2024-03-11T21:55:31.980248+0000","last_became_active":"2024-03-11T21:54:54.450842+0000","last_became_peered":"2024-03-11T21:54:54.450842+0000","last_unstale":"2024-03-11T21:55:31.980248+0000","last_undegraded":"2024-03-11T21:55:31.980248+0000","last_fullsized":"2024-03-11T21:55:31.980248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:02.413080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064964+0000","last_change":"2024-03-11T21:54:56.461045+0000","last_active":"2024-03-11T21:55:32.064964+0000","last_peered":"2024-03-11T21:55:32.064964+0000","last_clean":"2024-03-11T21:55:32.064964+0000","last_became_active":"2024-03-11T21:54:56.460761+0000","last_became_peered":"2024-03-11T21:54:56.460761+0000","last_unstale":"2024-03-11T21:55:32.064964+0000","last_undegraded":"2024-03-11T21:55:32.064964+0000","last_fullsized":"2024-03-11T21:55:32.064964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:40:53.333395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934263+0000","last_change":"2024-03-11T21:54:50.415035+0000","last_active":"2024-03-11T21:55:31.934263+0000","last_peered":"2024-03-11T21:55:31.934263+0000","last_clean":"2024-03-11T21:55:31.934263+0000","last_became_active":"2024-03-11T21:54:50.413522+0000","last_became_peered":"2024-03-11T21:54:50.413522+0000","last_unstale":"2024-03-11T21:55:31.934263+0000","last_undegraded":"2024-03-11T21:55:31.934263+0000","last_fullsized":"2024-03-11T21:55:31.934263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:58.564393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831128+0000","last_change":"2024-03-11T21:54:52.429688+0000","last_active":"2024-03-11T21:55:31.831128+0000","last_peered":"2024-03-11T21:55:31.831128+0000","last_clean":"2024-03-11T21:55:31.831128+0000","last_became_active":"2024-03-11T21:54:52.429356+0000","last_became_peered":"2024-03-11T21:54:52.429356+0000","last_unstale":"2024-03-11T21:55:31.831128+0000","last_undegraded":"2024-03-11T21:55:31.831128+0000","last_fullsized":"2024-03-11T21:55:31.831128+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:49:24.721134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.471946+0000","last_change":"2024-03-11T21:54:54.450815+0000","last_active":"2024-03-11T21:56:05.471946+0000","last_peered":"2024-03-11T21:56:05.471946+0000","last_clean":"2024-03-11T21:56:05.471946+0000","last_became_active":"2024-03-11T21:54:54.450407+0000","last_became_peered":"2024-03-11T21:54:54.450407+0000","last_unstale":"2024-03-11T21:56:05.471946+0000","last_undegraded":"2024-03-11T21:56:05.471946+0000","last_fullsized":"2024-03-11T21:56:05.471946+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:53:27.167368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946609+0000","last_change":"2024-03-11T21:54:56.536130+0000","last_active":"2024-03-11T21:55:31.946609+0000","last_peered":"2024-03-11T21:55:31.946609+0000","last_clean":"2024-03-11T21:55:31.946609+0000","last_became_active":"2024-03-11T21:54:56.535833+0000","last_became_peered":"2024-03-11T21:54:56.535833+0000","last_unstale":"2024-03-11T21:55:31.946609+0000","last_undegraded":"2024-03-11T21:55:31.946609+0000","last_fullsized":"2024-03-11T21:55:31.946609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:54:36.263864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066280+0000","last_change":"2024-03-11T21:54:50.417027+0000","last_active":"2024-03-11T21:55:32.066280+0000","last_peered":"2024-03-11T21:55:32.066280+0000","last_clean":"2024-03-11T21:55:32.066280+0000","last_became_active":"2024-03-11T21:54:50.416723+0000","last_became_peered":"2024-03-11T21:54:50.416723+0000","last_unstale":"2024-03-11T21:55:32.066280+0000","last_undegraded":"2024-03-11T21:55:32.066280+0000","last_fullsized":"2024-03-11T21:55:32.066280+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:17:12.360593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831125+0000","last_change":"2024-03-11T21:54:52.431466+0000","last_active":"2024-03-11T21:55:31.831125+0000","last_peered":"2024-03-11T21:55:31.831125+0000","last_clean":"2024-03-11T21:55:31.831125+0000","last_became_active":"2024-03-11T21:54:52.431151+0000","last_became_peered":"2024-03-11T21:54:52.431151+0000","last_unstale":"2024-03-11T21:55:31.831125+0000","last_undegraded":"2024-03-11T21:55:31.831125+0000","last_fullsized":"2024-03-11T21:55:31.831125+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:08.938448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.471972+0000","last_change":"2024-03-11T21:54:54.445374+0000","last_active":"2024-03-11T21:56:05.471972+0000","last_peered":"2024-03-11T21:56:05.471972+0000","last_clean":"2024-03-11T21:56:05.471972+0000","last_became_active":"2024-03-11T21:54:54.445070+0000","last_became_peered":"2024-03-11T21:54:54.445070+0000","last_unstale":"2024-03-11T21:56:05.471972+0000","last_undegraded":"2024-03-11T21:56:05.471972+0000","last_fullsized":"2024-03-11T21:56:05.471972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:38.796149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831972+0000","last_change":"2024-03-11T21:54:56.452272+0000","last_active":"2024-03-11T21:55:31.831972+0000","last_peered":"2024-03-11T21:55:31.831972+0000","last_clean":"2024-03-11T21:55:31.831972+0000","last_became_active":"2024-03-11T21:54:56.452077+0000","last_became_peered":"2024-03-11T21:54:56.452077+0000","last_unstale":"2024-03-11T21:55:31.831972+0000","last_undegraded":"2024-03-11T21:55:31.831972+0000","last_fullsized":"2024-03-11T21:55:31.831972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:40.718941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066769+0000","last_change":"2024-03-11T21:54:50.418599+0000","last_active":"2024-03-11T21:55:32.066769+0000","last_peered":"2024-03-11T21:55:32.066769+0000","last_clean":"2024-03-11T21:55:32.066769+0000","last_became_active":"2024-03-11T21:54:50.418290+0000","last_became_peered":"2024-03-11T21:54:50.418290+0000","last_unstale":"2024-03-11T21:55:32.066769+0000","last_undegraded":"2024-03-11T21:55:32.066769+0000","last_fullsized":"2024-03-11T21:55:32.066769+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:09:18.950583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065006+0000","last_change":"2024-03-11T21:54:52.436091+0000","last_active":"2024-03-11T21:55:32.065006+0000","last_peered":"2024-03-11T21:55:32.065006+0000","last_clean":"2024-03-11T21:55:32.065006+0000","last_became_active":"2024-03-11T21:54:52.433159+0000","last_became_peered":"2024-03-11T21:54:52.433159+0000","last_unstale":"2024-03-11T21:55:32.065006+0000","last_undegraded":"2024-03-11T21:55:32.065006+0000","last_fullsized":"2024-03-11T21:55:32.065006+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:17.575810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830966+0000","last_change":"2024-03-11T21:54:54.454078+0000","last_active":"2024-03-11T21:55:31.830966+0000","last_peered":"2024-03-11T21:55:31.830966+0000","last_clean":"2024-03-11T21:55:31.830966+0000","last_became_active":"2024-03-11T21:54:54.453731+0000","last_became_peered":"2024-03-11T21:54:54.453731+0000","last_unstale":"2024-03-11T21:55:31.830966+0000","last_undegraded":"2024-03-11T21:55:31.830966+0000","last_fullsized":"2024-03-11T21:55:31.830966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:53:08.384585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934086+0000","last_change":"2024-03-11T21:54:56.456245+0000","last_active":"2024-03-11T21:55:31.934086+0000","last_peered":"2024-03-11T21:55:31.934086+0000","last_clean":"2024-03-11T21:55:31.934086+0000","last_became_active":"2024-03-11T21:54:56.455948+0000","last_became_peered":"2024-03-11T21:54:56.455948+0000","last_unstale":"2024-03-11T21:55:31.934086+0000","last_undegraded":"2024-03-11T21:55:31.934086+0000","last_fullsized":"2024-03-11T21:55:31.934086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:42:06.952202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472174+0000","last_change":"2024-03-11T21:54:54.446693+0000","last_active":"2024-03-11T21:56:05.472174+0000","last_peered":"2024-03-11T21:56:05.472174+0000","last_clean":"2024-03-11T21:56:05.472174+0000","last_became_active":"2024-03-11T21:54:54.446397+0000","last_became_peered":"2024-03-11T21:54:54.446397+0000","last_unstale":"2024-03-11T21:56:05.472174+0000","last_undegraded":"2024-03-11T21:56:05.472174+0000","last_fullsized":"2024-03-11T21:56:05.472174+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:38.116626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064952+0000","last_change":"2024-03-11T21:54:50.413463+0000","last_active":"2024-03-11T21:55:32.064952+0000","last_peered":"2024-03-11T21:55:32.064952+0000","last_clean":"2024-03-11T21:55:32.064952+0000","last_became_active":"2024-03-11T21:54:50.413167+0000","last_became_peered":"2024-03-11T21:54:50.413167+0000","last_unstale":"2024-03-11T21:55:32.064952+0000","last_undegraded":"2024-03-11T21:55:32.064952+0000","last_fullsized":"2024-03-11T21:55:32.064952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:58.191595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831802+0000","last_change":"2024-03-11T21:54:52.434870+0000","last_active":"2024-03-11T21:55:31.831802+0000","last_peered":"2024-03-11T21:55:31.831802+0000","last_clean":"2024-03-11T21:55:31.831802+0000","last_became_active":"2024-03-11T21:54:52.434127+0000","last_became_peered":"2024-03-11T21:54:52.434127+0000","last_unstale":"2024-03-11T21:55:31.831802+0000","last_undegraded":"2024-03-11T21:55:31.831802+0000","last_fullsized":"2024-03-11T21:55:31.831802+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:55:35.588050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065867+0000","last_change":"2024-03-11T21:54:56.535576+0000","last_active":"2024-03-11T21:55:32.065867+0000","last_peered":"2024-03-11T21:55:32.065867+0000","last_clean":"2024-03-11T21:55:32.065867+0000","last_became_active":"2024-03-11T21:54:56.534987+0000","last_became_peered":"2024-03-11T21:54:56.534987+0000","last_unstale":"2024-03-11T21:55:32.065867+0000","last_undegraded":"2024-03-11T21:55:32.065867+0000","last_fullsized":"2024-03-11T21:55:32.065867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:57.107172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934499+0000","last_change":"2024-03-11T21:54:54.449099+0000","last_active":"2024-03-11T21:55:31.934499+0000","last_peered":"2024-03-11T21:55:31.934499+0000","last_clean":"2024-03-11T21:55:31.934499+0000","last_became_active":"2024-03-11T21:54:54.448782+0000","last_became_peered":"2024-03-11T21:54:54.448782+0000","last_unstale":"2024-03-11T21:55:31.934499+0000","last_undegraded":"2024-03-11T21:55:31.934499+0000","last_fullsized":"2024-03-11T21:55:31.934499+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:06:45.651830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831023+0000","last_change":"2024-03-11T21:54:50.415240+0000","last_active":"2024-03-11T21:55:31.831023+0000","last_peered":"2024-03-11T21:55:31.831023+0000","last_clean":"2024-03-11T21:55:31.831023+0000","last_became_active":"2024-03-11T21:54:50.414696+0000","last_became_peered":"2024-03-11T21:54:50.414696+0000","last_unstale":"2024-03-11T21:55:31.831023+0000","last_undegraded":"2024-03-11T21:55:31.831023+0000","last_fullsized":"2024-03-11T21:55:31.831023+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:11:24.610801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831332+0000","last_change":"2024-03-11T21:54:52.435284+0000","last_active":"2024-03-11T21:55:31.831332+0000","last_peered":"2024-03-11T21:55:31.831332+0000","last_clean":"2024-03-11T21:55:31.831332+0000","last_became_active":"2024-03-11T21:54:52.434854+0000","last_became_peered":"2024-03-11T21:54:52.434854+0000","last_unstale":"2024-03-11T21:55:31.831332+0000","last_undegraded":"2024-03-11T21:55:31.831332+0000","last_fullsized":"2024-03-11T21:55:31.831332+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:50:21.836802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832877+0000","last_change":"2024-03-11T21:54:56.461228+0000","last_active":"2024-03-11T21:55:31.832877+0000","last_peered":"2024-03-11T21:55:31.832877+0000","last_clean":"2024-03-11T21:55:31.832877+0000","last_became_active":"2024-03-11T21:54:56.460838+0000","last_became_peered":"2024-03-11T21:54:56.460838+0000","last_unstale":"2024-03-11T21:55:31.832877+0000","last_undegraded":"2024-03-11T21:55:31.832877+0000","last_fullsized":"2024-03-11T21:55:31.832877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:00.935613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831556+0000","last_change":"2024-03-11T21:54:54.448062+0000","last_active":"2024-03-11T21:55:31.831556+0000","last_peered":"2024-03-11T21:55:31.831556+0000","last_clean":"2024-03-11T21:55:31.831556+0000","last_became_active":"2024-03-11T21:54:54.447705+0000","last_became_peered":"2024-03-11T21:54:54.447705+0000","last_unstale":"2024-03-11T21:55:31.831556+0000","last_undegraded":"2024-03-11T21:55:31.831556+0000","last_fullsized":"2024-03-11T21:55:31.831556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:43:47.716024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.979996+0000","last_change":"2024-03-11T21:54:50.416934+0000","last_active":"2024-03-11T21:55:31.979996+0000","last_peered":"2024-03-11T21:55:31.979996+0000","last_clean":"2024-03-11T21:55:31.979996+0000","last_became_active":"2024-03-11T21:54:50.416640+0000","last_became_peered":"2024-03-11T21:54:50.416640+0000","last_unstale":"2024-03-11T21:55:31.979996+0000","last_undegraded":"2024-03-11T21:55:31.979996+0000","last_fullsized":"2024-03-11T21:55:31.979996+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:00:51.800411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831845+0000","last_change":"2024-03-11T21:54:52.433430+0000","last_active":"2024-03-11T21:55:31.831845+0000","last_peered":"2024-03-11T21:55:31.831845+0000","last_clean":"2024-03-11T21:55:31.831845+0000","last_became_active":"2024-03-11T21:54:52.433140+0000","last_became_peered":"2024-03-11T21:54:52.433140+0000","last_unstale":"2024-03-11T21:55:31.831845+0000","last_undegraded":"2024-03-11T21:55:31.831845+0000","last_fullsized":"2024-03-11T21:55:31.831845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:25:57.587156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831664+0000","last_change":"2024-03-11T21:54:56.448523+0000","last_active":"2024-03-11T21:55:31.831664+0000","last_peered":"2024-03-11T21:55:31.831664+0000","last_clean":"2024-03-11T21:55:31.831664+0000","last_became_active":"2024-03-11T21:54:56.448190+0000","last_became_peered":"2024-03-11T21:54:56.448190+0000","last_unstale":"2024-03-11T21:55:31.831664+0000","last_undegraded":"2024-03-11T21:55:31.831664+0000","last_fullsized":"2024-03-11T21:55:31.831664+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:06:12.068288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.471973+0000","last_change":"2024-03-11T21:54:54.448651+0000","last_active":"2024-03-11T21:56:05.471973+0000","last_peered":"2024-03-11T21:56:05.471973+0000","last_clean":"2024-03-11T21:56:05.471973+0000","last_became_active":"2024-03-11T21:54:54.448263+0000","last_became_peered":"2024-03-11T21:54:54.448263+0000","last_unstale":"2024-03-11T21:56:05.471973+0000","last_undegraded":"2024-03-11T21:56:05.471973+0000","last_fullsized":"2024-03-11T21:56:05.471973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:18.422914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065789+0000","last_change":"2024-03-11T21:54:50.419727+0000","last_active":"2024-03-11T21:55:32.065789+0000","last_peered":"2024-03-11T21:55:32.065789+0000","last_clean":"2024-03-11T21:55:32.065789+0000","last_became_active":"2024-03-11T21:54:50.419444+0000","last_became_peered":"2024-03-11T21:54:50.419444+0000","last_unstale":"2024-03-11T21:55:32.065789+0000","last_undegraded":"2024-03-11T21:55:32.065789+0000","last_fullsized":"2024-03-11T21:55:32.065789+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:25.570221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831573+0000","last_change":"2024-03-11T21:54:52.434062+0000","last_active":"2024-03-11T21:55:31.831573+0000","last_peered":"2024-03-11T21:55:31.831573+0000","last_clean":"2024-03-11T21:55:31.831573+0000","last_became_active":"2024-03-11T21:54:52.433752+0000","last_became_peered":"2024-03-11T21:54:52.433752+0000","last_unstale":"2024-03-11T21:55:31.831573+0000","last_undegraded":"2024-03-11T21:55:31.831573+0000","last_fullsized":"2024-03-11T21:55:31.831573+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:00:33.622518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201301,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28392,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712664,"statfs":{"total":95990841344,"available":95961767936,"internally_reserved":0,"allocated":1404928,"data_stored":926806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32532,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708524,"statfs":{"total":95990841344,"available":95957528576,"internally_reserved":0,"allocated":1384448,"data_stored":925755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713236,"statfs":{"total":95990841344,"available":95962353664,"internally_reserved":0,"allocated":819200,"data_stored":335885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28388,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712668,"statfs":{"total":95990841344,"available":95961772032,"internally_reserved":0,"allocated":1400832,"data_stored":925662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713256,"statfs":{"total":95990841344,"available":95962374144,"internally_reserved":0,"allocated":798720,"data_stored":335452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709072,"statfs":{"total":95990841344,"available":95958089728,"internally_reserved":0,"allocated":823296,"data_stored":335404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713240,"statfs":{"total":95990841344,"available":95962357760,"internally_reserved":0,"allocated":815104,"data_stored":335503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724576,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709088,"statfs":{"total":95990841344,"available":95958106112,"internally_reserved":0,"allocated":806912,"data_stored":336057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:56:16.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:56:17.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:16 smithi102 ceph-mon[25338]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:56:17.026 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:16 smithi102 ceph-mon[34177]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:56:17.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:16 smithi110 ceph-mon[30535]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:56:17.322 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 podman[78297]: 2024-03-11 21:56:16.937486191 +0000 UTC m=+0.026321813 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-11T21:56:17.322 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 podman[78297]: 2024-03-11 21:56:17.060546958 +0000 UTC m=+0.149382550 volume create 45e027222c09c11abb05eb11b05da8a3a926d94b6fe36a60a11b81a58841f7c3 2024-03-11T21:56:17.322 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 podman[78297]: 2024-03-11 21:56:17.169236395 +0000 UTC m=+0.258071988 container create c29d8bfd1c1f8a7d4ad021d7896e47cc6a6bbb637854d77c87ed8705d0cd76a8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:56:17.669 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 podman[78297]: 2024-03-11 21:56:17.471864619 +0000 UTC m=+0.560700212 container init c29d8bfd1c1f8a7d4ad021d7896e47cc6a6bbb637854d77c87ed8705d0cd76a8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:56:17.669 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 podman[78297]: 2024-03-11 21:56:17.474706165 +0000 UTC m=+0.563541759 container start c29d8bfd1c1f8a7d4ad021d7896e47cc6a6bbb637854d77c87ed8705d0cd76a8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:56:17.669 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:17.486Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T21:56:17.669 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:17.487Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T21:56:17.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:17.488Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.102 port=9094 2024-03-11T21:56:17.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:17.488Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T21:56:17.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:17.511Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:56:17.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:17.511Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:56:17.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:17.514Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T21:56:17.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:17.514Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T21:56:17.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 bash[78297]: c29d8bfd1c1f8a7d4ad021d7896e47cc6a6bbb637854d77c87ed8705d0cd76a8 2024-03-11T21:56:17.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:17 smithi102 systemd[1]: Started Ceph alertmanager.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:56:17.706 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:18.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:18.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: Deploying daemon grafana.a on smithi110 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[25338]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:18.087 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:17] ENGINE Bus STOPPING 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:17] ENGINE Bus STOPPED 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:17] ENGINE Bus STARTING 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:17] ENGINE Serving on http://:::9283 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:17] ENGINE Bus STARTED 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:56:18.089 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.089 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.089 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:56:18.089 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:56:18.089 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.089 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: Deploying daemon grafana.a on smithi110 2024-03-11T21:56:18.089 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:17 smithi102 ceph-mon[34177]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:18.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:18.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: Deploying daemon grafana.a on smithi110 2024-03-11T21:56:18.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:17 smithi110 ceph-mon[30535]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:19.581 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:56:19.583 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-11T21:56:19.901 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:19 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:19.489Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.0002381s 2024-03-11T21:56:20.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:19 smithi102 ceph-mon[25338]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:56:20.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:19 smithi102 ceph-mon[25338]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:20.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:19 smithi102 ceph-mon[34177]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:56:20.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:19 smithi102 ceph-mon[34177]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:20.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:19 smithi110 ceph-mon[30535]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:56:20.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:19 smithi110 ceph-mon[30535]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:20.370 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-11T21:56:17.849487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1039,"num_read_kb":882,"num_write":681,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":661,"ondisk_log_size":661,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":236700,"kb_used_data":8060,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749691748,"statfs":{"total":767926730752,"available":767684349952,"internally_reserved":0,"allocated":8253440,"data_stored":4456524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002589"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831881+0000","last_change":"2024-03-11T21:54:56.449478+0000","last_active":"2024-03-11T21:55:31.831881+0000","last_peered":"2024-03-11T21:55:31.831881+0000","last_clean":"2024-03-11T21:55:31.831881+0000","last_became_active":"2024-03-11T21:54:56.449175+0000","last_became_peered":"2024-03-11T21:54:56.449175+0000","last_unstale":"2024-03-11T21:55:31.831881+0000","last_undegraded":"2024-03-11T21:55:31.831881+0000","last_fullsized":"2024-03-11T21:55:31.831881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:42:38.581508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830654+0000","last_change":"2024-03-11T21:54:54.449954+0000","last_active":"2024-03-11T21:55:31.830654+0000","last_peered":"2024-03-11T21:55:31.830654+0000","last_clean":"2024-03-11T21:55:31.830654+0000","last_became_active":"2024-03-11T21:54:54.449654+0000","last_became_peered":"2024-03-11T21:54:54.449654+0000","last_unstale":"2024-03-11T21:55:31.830654+0000","last_undegraded":"2024-03-11T21:55:31.830654+0000","last_fullsized":"2024-03-11T21:55:31.830654+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:35.109872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065812+0000","last_change":"2024-03-11T21:54:50.419611+0000","last_active":"2024-03-11T21:55:32.065812+0000","last_peered":"2024-03-11T21:55:32.065812+0000","last_clean":"2024-03-11T21:55:32.065812+0000","last_became_active":"2024-03-11T21:54:50.419123+0000","last_became_peered":"2024-03-11T21:54:50.419123+0000","last_unstale":"2024-03-11T21:55:32.065812+0000","last_undegraded":"2024-03-11T21:55:32.065812+0000","last_fullsized":"2024-03-11T21:55:32.065812+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:41:49.213574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832038+0000","last_change":"2024-03-11T21:55:13.514481+0000","last_active":"2024-03-11T21:55:31.832038+0000","last_peered":"2024-03-11T21:55:31.832038+0000","last_clean":"2024-03-11T21:55:31.832038+0000","last_became_active":"2024-03-11T21:55:13.508420+0000","last_became_peered":"2024-03-11T21:55:13.508420+0000","last_unstale":"2024-03-11T21:55:31.832038+0000","last_undegraded":"2024-03-11T21:55:31.832038+0000","last_fullsized":"2024-03-11T21:55:31.832038+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:21:57.498731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065699+0000","last_change":"2024-03-11T21:54:56.537559+0000","last_active":"2024-03-11T21:55:32.065699+0000","last_peered":"2024-03-11T21:55:32.065699+0000","last_clean":"2024-03-11T21:55:32.065699+0000","last_became_active":"2024-03-11T21:54:56.537217+0000","last_became_peered":"2024-03-11T21:54:56.537217+0000","last_unstale":"2024-03-11T21:55:32.065699+0000","last_undegraded":"2024-03-11T21:55:32.065699+0000","last_fullsized":"2024-03-11T21:55:32.065699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:53:16.201491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830944+0000","last_change":"2024-03-11T21:54:54.445651+0000","last_active":"2024-03-11T21:55:31.830944+0000","last_peered":"2024-03-11T21:55:31.830944+0000","last_clean":"2024-03-11T21:55:31.830944+0000","last_became_active":"2024-03-11T21:54:54.445342+0000","last_became_peered":"2024-03-11T21:54:54.445342+0000","last_unstale":"2024-03-11T21:55:31.830944+0000","last_undegraded":"2024-03-11T21:55:31.830944+0000","last_fullsized":"2024-03-11T21:55:31.830944+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:15:59.165041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066715+0000","last_change":"2024-03-11T21:54:50.413129+0000","last_active":"2024-03-11T21:55:32.066715+0000","last_peered":"2024-03-11T21:55:32.066715+0000","last_clean":"2024-03-11T21:55:32.066715+0000","last_became_active":"2024-03-11T21:54:50.412818+0000","last_became_peered":"2024-03-11T21:54:50.412818+0000","last_unstale":"2024-03-11T21:55:32.066715+0000","last_undegraded":"2024-03-11T21:55:32.066715+0000","last_fullsized":"2024-03-11T21:55:32.066715+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:10.340260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065828+0000","last_change":"2024-03-11T21:54:52.438782+0000","last_active":"2024-03-11T21:55:32.065828+0000","last_peered":"2024-03-11T21:55:32.065828+0000","last_clean":"2024-03-11T21:55:32.065828+0000","last_became_active":"2024-03-11T21:54:52.438472+0000","last_became_peered":"2024-03-11T21:54:52.438472+0000","last_unstale":"2024-03-11T21:55:32.065828+0000","last_undegraded":"2024-03-11T21:55:32.065828+0000","last_fullsized":"2024-03-11T21:55:32.065828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:43.708792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065986+0000","last_change":"2024-03-11T21:54:56.456252+0000","last_active":"2024-03-11T21:55:32.065986+0000","last_peered":"2024-03-11T21:55:32.065986+0000","last_clean":"2024-03-11T21:55:32.065986+0000","last_became_active":"2024-03-11T21:54:56.455938+0000","last_became_peered":"2024-03-11T21:54:56.455938+0000","last_unstale":"2024-03-11T21:55:32.065986+0000","last_undegraded":"2024-03-11T21:55:32.065986+0000","last_fullsized":"2024-03-11T21:55:32.065986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:38:27.060415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933966+0000","last_change":"2024-03-11T21:54:54.455992+0000","last_active":"2024-03-11T21:55:31.933966+0000","last_peered":"2024-03-11T21:55:31.933966+0000","last_clean":"2024-03-11T21:55:31.933966+0000","last_became_active":"2024-03-11T21:54:54.455634+0000","last_became_peered":"2024-03-11T21:54:54.455634+0000","last_unstale":"2024-03-11T21:55:31.933966+0000","last_undegraded":"2024-03-11T21:55:31.933966+0000","last_fullsized":"2024-03-11T21:55:31.933966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:45.875406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934060+0000","last_change":"2024-03-11T21:54:50.418368+0000","last_active":"2024-03-11T21:55:31.934060+0000","last_peered":"2024-03-11T21:55:31.934060+0000","last_clean":"2024-03-11T21:55:31.934060+0000","last_became_active":"2024-03-11T21:54:50.418011+0000","last_became_peered":"2024-03-11T21:54:50.418011+0000","last_unstale":"2024-03-11T21:55:31.934060+0000","last_undegraded":"2024-03-11T21:55:31.934060+0000","last_fullsized":"2024-03-11T21:55:31.934060+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:03.100681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832113+0000","last_change":"2024-03-11T21:54:52.433188+0000","last_active":"2024-03-11T21:55:31.832113+0000","last_peered":"2024-03-11T21:55:31.832113+0000","last_clean":"2024-03-11T21:55:31.832113+0000","last_became_active":"2024-03-11T21:54:52.432859+0000","last_became_peered":"2024-03-11T21:54:52.432859+0000","last_unstale":"2024-03-11T21:55:31.832113+0000","last_undegraded":"2024-03-11T21:55:31.832113+0000","last_fullsized":"2024-03-11T21:55:31.832113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:15:39.303127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933699+0000","last_change":"2024-03-11T21:54:56.463176+0000","last_active":"2024-03-11T21:55:31.933699+0000","last_peered":"2024-03-11T21:55:31.933699+0000","last_clean":"2024-03-11T21:55:31.933699+0000","last_became_active":"2024-03-11T21:54:56.462892+0000","last_became_peered":"2024-03-11T21:54:56.462892+0000","last_unstale":"2024-03-11T21:55:31.933699+0000","last_undegraded":"2024-03-11T21:55:31.933699+0000","last_fullsized":"2024-03-11T21:55:31.933699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:31.708144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946369+0000","last_change":"2024-03-11T21:54:54.455379+0000","last_active":"2024-03-11T21:55:31.946369+0000","last_peered":"2024-03-11T21:55:31.946369+0000","last_clean":"2024-03-11T21:55:31.946369+0000","last_became_active":"2024-03-11T21:54:54.455073+0000","last_became_peered":"2024-03-11T21:54:54.455073+0000","last_unstale":"2024-03-11T21:55:31.946369+0000","last_undegraded":"2024-03-11T21:55:31.946369+0000","last_fullsized":"2024-03-11T21:55:31.946369+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:30.235270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934133+0000","last_change":"2024-03-11T21:54:50.410817+0000","last_active":"2024-03-11T21:55:31.934133+0000","last_peered":"2024-03-11T21:55:31.934133+0000","last_clean":"2024-03-11T21:55:31.934133+0000","last_became_active":"2024-03-11T21:54:50.410241+0000","last_became_peered":"2024-03-11T21:54:50.410241+0000","last_unstale":"2024-03-11T21:55:31.934133+0000","last_undegraded":"2024-03-11T21:55:31.934133+0000","last_fullsized":"2024-03-11T21:55:31.934133+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:49.365681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831637+0000","last_change":"2024-03-11T21:54:52.438105+0000","last_active":"2024-03-11T21:55:31.831637+0000","last_peered":"2024-03-11T21:55:31.831637+0000","last_clean":"2024-03-11T21:55:31.831637+0000","last_became_active":"2024-03-11T21:54:52.437744+0000","last_became_peered":"2024-03-11T21:54:52.437744+0000","last_unstale":"2024-03-11T21:55:31.831637+0000","last_undegraded":"2024-03-11T21:55:31.831637+0000","last_fullsized":"2024-03-11T21:55:31.831637+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:37.745150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831463+0000","last_change":"2024-03-11T21:54:56.537012+0000","last_active":"2024-03-11T21:55:31.831463+0000","last_peered":"2024-03-11T21:55:31.831463+0000","last_clean":"2024-03-11T21:55:31.831463+0000","last_became_active":"2024-03-11T21:54:56.536644+0000","last_became_peered":"2024-03-11T21:54:56.536644+0000","last_unstale":"2024-03-11T21:55:31.831463+0000","last_undegraded":"2024-03-11T21:55:31.831463+0000","last_fullsized":"2024-03-11T21:55:31.831463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:57.627059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831565+0000","last_change":"2024-03-11T21:54:54.445955+0000","last_active":"2024-03-11T21:55:31.831565+0000","last_peered":"2024-03-11T21:55:31.831565+0000","last_clean":"2024-03-11T21:55:31.831565+0000","last_became_active":"2024-03-11T21:54:54.445624+0000","last_became_peered":"2024-03-11T21:54:54.445624+0000","last_unstale":"2024-03-11T21:55:31.831565+0000","last_undegraded":"2024-03-11T21:55:31.831565+0000","last_fullsized":"2024-03-11T21:55:31.831565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:57:38.628568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066562+0000","last_change":"2024-03-11T21:54:50.422131+0000","last_active":"2024-03-11T21:55:32.066562+0000","last_peered":"2024-03-11T21:55:32.066562+0000","last_clean":"2024-03-11T21:55:32.066562+0000","last_became_active":"2024-03-11T21:54:50.421356+0000","last_became_peered":"2024-03-11T21:54:50.421356+0000","last_unstale":"2024-03-11T21:55:32.066562+0000","last_undegraded":"2024-03-11T21:55:32.066562+0000","last_fullsized":"2024-03-11T21:55:32.066562+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:14:21.895413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832590+0000","last_change":"2024-03-11T21:54:52.432518+0000","last_active":"2024-03-11T21:55:31.832590+0000","last_peered":"2024-03-11T21:55:31.832590+0000","last_clean":"2024-03-11T21:55:31.832590+0000","last_became_active":"2024-03-11T21:54:52.432140+0000","last_became_peered":"2024-03-11T21:54:52.432140+0000","last_unstale":"2024-03-11T21:55:31.832590+0000","last_undegraded":"2024-03-11T21:55:31.832590+0000","last_fullsized":"2024-03-11T21:55:31.832590+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:51:11.877624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066529+0000","last_change":"2024-03-11T21:54:56.536166+0000","last_active":"2024-03-11T21:55:32.066529+0000","last_peered":"2024-03-11T21:55:32.066529+0000","last_clean":"2024-03-11T21:55:32.066529+0000","last_became_active":"2024-03-11T21:54:56.535015+0000","last_became_peered":"2024-03-11T21:54:56.535015+0000","last_unstale":"2024-03-11T21:55:32.066529+0000","last_undegraded":"2024-03-11T21:55:32.066529+0000","last_fullsized":"2024-03-11T21:55:32.066529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:02:49.161746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831471+0000","last_change":"2024-03-11T21:54:54.453732+0000","last_active":"2024-03-11T21:55:31.831471+0000","last_peered":"2024-03-11T21:55:31.831471+0000","last_clean":"2024-03-11T21:55:31.831471+0000","last_became_active":"2024-03-11T21:54:54.453333+0000","last_became_peered":"2024-03-11T21:54:54.453333+0000","last_unstale":"2024-03-11T21:55:31.831471+0000","last_undegraded":"2024-03-11T21:55:31.831471+0000","last_fullsized":"2024-03-11T21:55:31.831471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:34.135236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832230+0000","last_change":"2024-03-11T21:54:50.412648+0000","last_active":"2024-03-11T21:55:31.832230+0000","last_peered":"2024-03-11T21:55:31.832230+0000","last_clean":"2024-03-11T21:55:31.832230+0000","last_became_active":"2024-03-11T21:54:50.410393+0000","last_became_peered":"2024-03-11T21:54:50.410393+0000","last_unstale":"2024-03-11T21:55:31.832230+0000","last_undegraded":"2024-03-11T21:55:31.832230+0000","last_fullsized":"2024-03-11T21:55:31.832230+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:41:50.324281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066283+0000","last_change":"2024-03-11T21:54:52.436246+0000","last_active":"2024-03-11T21:55:32.066283+0000","last_peered":"2024-03-11T21:55:32.066283+0000","last_clean":"2024-03-11T21:55:32.066283+0000","last_became_active":"2024-03-11T21:54:52.434955+0000","last_became_peered":"2024-03-11T21:54:52.434955+0000","last_unstale":"2024-03-11T21:55:32.066283+0000","last_undegraded":"2024-03-11T21:55:32.066283+0000","last_fullsized":"2024-03-11T21:55:32.066283+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:37:26.945043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946549+0000","last_change":"2024-03-11T21:54:56.456693+0000","last_active":"2024-03-11T21:55:31.946549+0000","last_peered":"2024-03-11T21:55:31.946549+0000","last_clean":"2024-03-11T21:55:31.946549+0000","last_became_active":"2024-03-11T21:54:56.456371+0000","last_became_peered":"2024-03-11T21:54:56.456371+0000","last_unstale":"2024-03-11T21:55:31.946549+0000","last_undegraded":"2024-03-11T21:55:31.946549+0000","last_fullsized":"2024-03-11T21:55:31.946549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:24:41.760797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472140+0000","last_change":"2024-03-11T21:54:54.448752+0000","last_active":"2024-03-11T21:56:05.472140+0000","last_peered":"2024-03-11T21:56:05.472140+0000","last_clean":"2024-03-11T21:56:05.472140+0000","last_became_active":"2024-03-11T21:54:54.448398+0000","last_became_peered":"2024-03-11T21:54:54.448398+0000","last_unstale":"2024-03-11T21:56:05.472140+0000","last_undegraded":"2024-03-11T21:56:05.472140+0000","last_fullsized":"2024-03-11T21:56:05.472140+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:03:20.410444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830787+0000","last_change":"2024-03-11T21:54:50.412242+0000","last_active":"2024-03-11T21:55:31.830787+0000","last_peered":"2024-03-11T21:55:31.830787+0000","last_clean":"2024-03-11T21:55:31.830787+0000","last_became_active":"2024-03-11T21:54:50.411916+0000","last_became_peered":"2024-03-11T21:54:50.411916+0000","last_unstale":"2024-03-11T21:55:31.830787+0000","last_undegraded":"2024-03-11T21:55:31.830787+0000","last_fullsized":"2024-03-11T21:55:31.830787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:37:15.981551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066396+0000","last_change":"2024-03-11T21:54:52.433086+0000","last_active":"2024-03-11T21:55:32.066396+0000","last_peered":"2024-03-11T21:55:32.066396+0000","last_clean":"2024-03-11T21:55:32.066396+0000","last_became_active":"2024-03-11T21:54:52.432691+0000","last_became_peered":"2024-03-11T21:54:52.432691+0000","last_unstale":"2024-03-11T21:55:32.066396+0000","last_undegraded":"2024-03-11T21:55:32.066396+0000","last_fullsized":"2024-03-11T21:55:32.066396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:48:57.755182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831727+0000","last_change":"2024-03-11T21:54:56.451709+0000","last_active":"2024-03-11T21:55:31.831727+0000","last_peered":"2024-03-11T21:55:31.831727+0000","last_clean":"2024-03-11T21:55:31.831727+0000","last_became_active":"2024-03-11T21:54:56.451369+0000","last_became_peered":"2024-03-11T21:54:56.451369+0000","last_unstale":"2024-03-11T21:55:31.831727+0000","last_undegraded":"2024-03-11T21:55:31.831727+0000","last_fullsized":"2024-03-11T21:55:31.831727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:42.284323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066581+0000","last_change":"2024-03-11T21:54:54.449788+0000","last_active":"2024-03-11T21:55:32.066581+0000","last_peered":"2024-03-11T21:55:32.066581+0000","last_clean":"2024-03-11T21:55:32.066581+0000","last_became_active":"2024-03-11T21:54:54.449461+0000","last_became_peered":"2024-03-11T21:54:54.449461+0000","last_unstale":"2024-03-11T21:55:32.066581+0000","last_undegraded":"2024-03-11T21:55:32.066581+0000","last_fullsized":"2024-03-11T21:55:32.066581+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:55:53.595676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064816+0000","last_change":"2024-03-11T21:54:50.418827+0000","last_active":"2024-03-11T21:55:32.064816+0000","last_peered":"2024-03-11T21:55:32.064816+0000","last_clean":"2024-03-11T21:55:32.064816+0000","last_became_active":"2024-03-11T21:54:50.418535+0000","last_became_peered":"2024-03-11T21:54:50.418535+0000","last_unstale":"2024-03-11T21:55:32.064816+0000","last_undegraded":"2024-03-11T21:55:32.064816+0000","last_fullsized":"2024-03-11T21:55:32.064816+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:04:07.214347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.980544+0000","last_change":"2024-03-11T21:54:52.434361+0000","last_active":"2024-03-11T21:55:31.980544+0000","last_peered":"2024-03-11T21:55:31.980544+0000","last_clean":"2024-03-11T21:55:31.980544+0000","last_became_active":"2024-03-11T21:54:52.433631+0000","last_became_peered":"2024-03-11T21:54:52.433631+0000","last_unstale":"2024-03-11T21:55:31.980544+0000","last_undegraded":"2024-03-11T21:55:31.980544+0000","last_fullsized":"2024-03-11T21:55:31.980544+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:56:35.530845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946341+0000","last_change":"2024-03-11T21:54:56.456134+0000","last_active":"2024-03-11T21:55:31.946341+0000","last_peered":"2024-03-11T21:55:31.946341+0000","last_clean":"2024-03-11T21:55:31.946341+0000","last_became_active":"2024-03-11T21:54:56.455755+0000","last_became_peered":"2024-03-11T21:54:56.455755+0000","last_unstale":"2024-03-11T21:55:31.946341+0000","last_undegraded":"2024-03-11T21:55:31.946341+0000","last_fullsized":"2024-03-11T21:55:31.946341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:53:19.091447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.979902+0000","last_change":"2024-03-11T21:54:54.448627+0000","last_active":"2024-03-11T21:55:31.979902+0000","last_peered":"2024-03-11T21:55:31.979902+0000","last_clean":"2024-03-11T21:55:31.979902+0000","last_became_active":"2024-03-11T21:54:54.448314+0000","last_became_peered":"2024-03-11T21:54:54.448314+0000","last_unstale":"2024-03-11T21:55:31.979902+0000","last_undegraded":"2024-03-11T21:55:31.979902+0000","last_fullsized":"2024-03-11T21:55:31.979902+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:58:07.417041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065655+0000","last_change":"2024-03-11T21:54:50.421248+0000","last_active":"2024-03-11T21:55:32.065655+0000","last_peered":"2024-03-11T21:55:32.065655+0000","last_clean":"2024-03-11T21:55:32.065655+0000","last_became_active":"2024-03-11T21:54:50.420906+0000","last_became_peered":"2024-03-11T21:54:50.420906+0000","last_unstale":"2024-03-11T21:55:32.065655+0000","last_undegraded":"2024-03-11T21:55:32.065655+0000","last_fullsized":"2024-03-11T21:55:32.065655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:23.113143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064893+0000","last_change":"2024-03-11T21:54:52.437893+0000","last_active":"2024-03-11T21:55:32.064893+0000","last_peered":"2024-03-11T21:55:32.064893+0000","last_clean":"2024-03-11T21:55:32.064893+0000","last_became_active":"2024-03-11T21:54:52.437581+0000","last_became_peered":"2024-03-11T21:54:52.437581+0000","last_unstale":"2024-03-11T21:55:32.064893+0000","last_undegraded":"2024-03-11T21:55:32.064893+0000","last_fullsized":"2024-03-11T21:55:32.064893+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:04:39.231991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065520+0000","last_change":"2024-03-11T21:54:56.536583+0000","last_active":"2024-03-11T21:55:32.065520+0000","last_peered":"2024-03-11T21:55:32.065520+0000","last_clean":"2024-03-11T21:55:32.065520+0000","last_became_active":"2024-03-11T21:54:56.536202+0000","last_became_peered":"2024-03-11T21:54:56.536202+0000","last_unstale":"2024-03-11T21:55:32.065520+0000","last_undegraded":"2024-03-11T21:55:32.065520+0000","last_fullsized":"2024-03-11T21:55:32.065520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:49.413728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946524+0000","last_change":"2024-03-11T21:54:54.447995+0000","last_active":"2024-03-11T21:55:31.946524+0000","last_peered":"2024-03-11T21:55:31.946524+0000","last_clean":"2024-03-11T21:55:31.946524+0000","last_became_active":"2024-03-11T21:54:54.446803+0000","last_became_peered":"2024-03-11T21:54:54.446803+0000","last_unstale":"2024-03-11T21:55:31.946524+0000","last_undegraded":"2024-03-11T21:55:31.946524+0000","last_fullsized":"2024-03-11T21:55:31.946524+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:57:22.943370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934183+0000","last_change":"2024-03-11T21:54:50.412364+0000","last_active":"2024-03-11T21:55:31.934183+0000","last_peered":"2024-03-11T21:55:31.934183+0000","last_clean":"2024-03-11T21:55:31.934183+0000","last_became_active":"2024-03-11T21:54:50.411181+0000","last_became_peered":"2024-03-11T21:54:50.411181+0000","last_unstale":"2024-03-11T21:55:31.934183+0000","last_undegraded":"2024-03-11T21:55:31.934183+0000","last_fullsized":"2024-03-11T21:55:31.934183+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:07.041244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065672+0000","last_change":"2024-03-11T21:54:52.438828+0000","last_active":"2024-03-11T21:55:32.065672+0000","last_peered":"2024-03-11T21:55:32.065672+0000","last_clean":"2024-03-11T21:55:32.065672+0000","last_became_active":"2024-03-11T21:54:52.438187+0000","last_became_peered":"2024-03-11T21:54:52.438187+0000","last_unstale":"2024-03-11T21:55:32.065672+0000","last_undegraded":"2024-03-11T21:55:32.065672+0000","last_fullsized":"2024-03-11T21:55:32.065672+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:12:09.221931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064854+0000","last_change":"2024-03-11T21:54:56.458260+0000","last_active":"2024-03-11T21:55:32.064854+0000","last_peered":"2024-03-11T21:55:32.064854+0000","last_clean":"2024-03-11T21:55:32.064854+0000","last_became_active":"2024-03-11T21:54:56.457458+0000","last_became_peered":"2024-03-11T21:54:56.457458+0000","last_unstale":"2024-03-11T21:55:32.064854+0000","last_undegraded":"2024-03-11T21:55:32.064854+0000","last_fullsized":"2024-03-11T21:55:32.064854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:22:19.422414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065704+0000","last_change":"2024-03-11T21:54:54.447062+0000","last_active":"2024-03-11T21:55:32.065704+0000","last_peered":"2024-03-11T21:55:32.065704+0000","last_clean":"2024-03-11T21:55:32.065704+0000","last_became_active":"2024-03-11T21:54:54.446750+0000","last_became_peered":"2024-03-11T21:54:54.446750+0000","last_unstale":"2024-03-11T21:55:32.065704+0000","last_undegraded":"2024-03-11T21:55:32.065704+0000","last_fullsized":"2024-03-11T21:55:32.065704+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:17:28.010411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946671+0000","last_change":"2024-03-11T21:54:50.420855+0000","last_active":"2024-03-11T21:55:31.946671+0000","last_peered":"2024-03-11T21:55:31.946671+0000","last_clean":"2024-03-11T21:55:31.946671+0000","last_became_active":"2024-03-11T21:54:50.420404+0000","last_became_peered":"2024-03-11T21:54:50.420404+0000","last_unstale":"2024-03-11T21:55:31.946671+0000","last_undegraded":"2024-03-11T21:55:31.946671+0000","last_fullsized":"2024-03-11T21:55:31.946671+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:25:28.405026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831247+0000","last_change":"2024-03-11T21:54:52.432141+0000","last_active":"2024-03-11T21:55:31.831247+0000","last_peered":"2024-03-11T21:55:31.831247+0000","last_clean":"2024-03-11T21:55:31.831247+0000","last_became_active":"2024-03-11T21:54:52.431667+0000","last_became_peered":"2024-03-11T21:54:52.431667+0000","last_unstale":"2024-03-11T21:55:31.831247+0000","last_undegraded":"2024-03-11T21:55:31.831247+0000","last_fullsized":"2024-03-11T21:55:31.831247+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:39.141139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064911+0000","last_change":"2024-03-11T21:54:56.462182+0000","last_active":"2024-03-11T21:55:32.064911+0000","last_peered":"2024-03-11T21:55:32.064911+0000","last_clean":"2024-03-11T21:55:32.064911+0000","last_became_active":"2024-03-11T21:54:56.460594+0000","last_became_peered":"2024-03-11T21:54:56.460594+0000","last_unstale":"2024-03-11T21:55:32.064911+0000","last_undegraded":"2024-03-11T21:55:32.064911+0000","last_fullsized":"2024-03-11T21:55:32.064911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:26.691208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831035+0000","last_change":"2024-03-11T21:54:54.447234+0000","last_active":"2024-03-11T21:55:31.831035+0000","last_peered":"2024-03-11T21:55:31.831035+0000","last_clean":"2024-03-11T21:55:31.831035+0000","last_became_active":"2024-03-11T21:54:54.446915+0000","last_became_peered":"2024-03-11T21:54:54.446915+0000","last_unstale":"2024-03-11T21:55:31.831035+0000","last_undegraded":"2024-03-11T21:55:31.831035+0000","last_fullsized":"2024-03-11T21:55:31.831035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:14.615758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831765+0000","last_change":"2024-03-11T21:54:50.415621+0000","last_active":"2024-03-11T21:55:31.831765+0000","last_peered":"2024-03-11T21:55:31.831765+0000","last_clean":"2024-03-11T21:55:31.831765+0000","last_became_active":"2024-03-11T21:54:50.414331+0000","last_became_peered":"2024-03-11T21:54:50.414331+0000","last_unstale":"2024-03-11T21:55:31.831765+0000","last_undegraded":"2024-03-11T21:55:31.831765+0000","last_fullsized":"2024-03-11T21:55:31.831765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:34:44.860393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831375+0000","last_change":"2024-03-11T21:54:52.434309+0000","last_active":"2024-03-11T21:55:31.831375+0000","last_peered":"2024-03-11T21:55:31.831375+0000","last_clean":"2024-03-11T21:55:31.831375+0000","last_became_active":"2024-03-11T21:54:52.434033+0000","last_became_peered":"2024-03-11T21:54:52.434033+0000","last_unstale":"2024-03-11T21:55:31.831375+0000","last_undegraded":"2024-03-11T21:55:31.831375+0000","last_fullsized":"2024-03-11T21:55:31.831375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:19.155542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066094+0000","last_change":"2024-03-11T21:54:56.462310+0000","last_active":"2024-03-11T21:55:32.066094+0000","last_peered":"2024-03-11T21:55:32.066094+0000","last_clean":"2024-03-11T21:55:32.066094+0000","last_became_active":"2024-03-11T21:54:56.461998+0000","last_became_peered":"2024-03-11T21:54:56.461998+0000","last_unstale":"2024-03-11T21:55:32.066094+0000","last_undegraded":"2024-03-11T21:55:32.066094+0000","last_fullsized":"2024-03-11T21:55:32.066094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:51.931704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472185+0000","last_change":"2024-03-11T21:54:54.450939+0000","last_active":"2024-03-11T21:56:05.472185+0000","last_peered":"2024-03-11T21:56:05.472185+0000","last_clean":"2024-03-11T21:56:05.472185+0000","last_became_active":"2024-03-11T21:54:54.450608+0000","last_became_peered":"2024-03-11T21:54:54.450608+0000","last_unstale":"2024-03-11T21:56:05.472185+0000","last_undegraded":"2024-03-11T21:56:05.472185+0000","last_fullsized":"2024-03-11T21:56:05.472185+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:58:39.219918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946500+0000","last_change":"2024-03-11T21:54:50.414322+0000","last_active":"2024-03-11T21:55:31.946500+0000","last_peered":"2024-03-11T21:55:31.946500+0000","last_clean":"2024-03-11T21:55:31.946500+0000","last_became_active":"2024-03-11T21:54:50.413934+0000","last_became_peered":"2024-03-11T21:54:50.413934+0000","last_unstale":"2024-03-11T21:55:31.946500+0000","last_undegraded":"2024-03-11T21:55:31.946500+0000","last_fullsized":"2024-03-11T21:55:31.946500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:51.320184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065957+0000","last_change":"2024-03-11T21:54:52.436118+0000","last_active":"2024-03-11T21:55:32.065957+0000","last_peered":"2024-03-11T21:55:32.065957+0000","last_clean":"2024-03-11T21:55:32.065957+0000","last_became_active":"2024-03-11T21:54:52.434857+0000","last_became_peered":"2024-03-11T21:54:52.434857+0000","last_unstale":"2024-03-11T21:55:32.065957+0000","last_undegraded":"2024-03-11T21:55:32.065957+0000","last_fullsized":"2024-03-11T21:55:32.065957+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:46:52.183401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831706+0000","last_change":"2024-03-11T21:54:56.450231+0000","last_active":"2024-03-11T21:55:31.831706+0000","last_peered":"2024-03-11T21:55:31.831706+0000","last_clean":"2024-03-11T21:55:31.831706+0000","last_became_active":"2024-03-11T21:54:56.449467+0000","last_became_peered":"2024-03-11T21:54:56.449467+0000","last_unstale":"2024-03-11T21:55:31.831706+0000","last_undegraded":"2024-03-11T21:55:31.831706+0000","last_fullsized":"2024-03-11T21:55:31.831706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:06.062812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472005+0000","last_change":"2024-03-11T21:54:54.452707+0000","last_active":"2024-03-11T21:56:05.472005+0000","last_peered":"2024-03-11T21:56:05.472005+0000","last_clean":"2024-03-11T21:56:05.472005+0000","last_became_active":"2024-03-11T21:54:54.452185+0000","last_became_peered":"2024-03-11T21:54:54.452185+0000","last_unstale":"2024-03-11T21:56:05.472005+0000","last_undegraded":"2024-03-11T21:56:05.472005+0000","last_fullsized":"2024-03-11T21:56:05.472005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:02:49.662461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065259+0000","last_change":"2024-03-11T21:54:50.421489+0000","last_active":"2024-03-11T21:55:32.065259+0000","last_peered":"2024-03-11T21:55:32.065259+0000","last_clean":"2024-03-11T21:55:32.065259+0000","last_became_active":"2024-03-11T21:54:50.420942+0000","last_became_peered":"2024-03-11T21:54:50.420942+0000","last_unstale":"2024-03-11T21:55:32.065259+0000","last_undegraded":"2024-03-11T21:55:32.065259+0000","last_fullsized":"2024-03-11T21:55:32.065259+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:50:36.980863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934369+0000","last_change":"2024-03-11T21:54:52.433167+0000","last_active":"2024-03-11T21:55:31.934369+0000","last_peered":"2024-03-11T21:55:31.934369+0000","last_clean":"2024-03-11T21:55:31.934369+0000","last_became_active":"2024-03-11T21:54:52.432837+0000","last_became_peered":"2024-03-11T21:54:52.432837+0000","last_unstale":"2024-03-11T21:55:31.934369+0000","last_undegraded":"2024-03-11T21:55:31.934369+0000","last_fullsized":"2024-03-11T21:55:31.934369+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:35:05.309569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832789+0000","last_change":"2024-03-11T21:54:56.453995+0000","last_active":"2024-03-11T21:55:31.832789+0000","last_peered":"2024-03-11T21:55:31.832789+0000","last_clean":"2024-03-11T21:55:31.832789+0000","last_became_active":"2024-03-11T21:54:56.453694+0000","last_became_peered":"2024-03-11T21:54:56.453694+0000","last_unstale":"2024-03-11T21:55:31.832789+0000","last_undegraded":"2024-03-11T21:55:31.832789+0000","last_fullsized":"2024-03-11T21:55:31.832789+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:42.432711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066825+0000","last_change":"2024-03-11T21:54:54.450613+0000","last_active":"2024-03-11T21:55:32.066825+0000","last_peered":"2024-03-11T21:55:32.066825+0000","last_clean":"2024-03-11T21:55:32.066825+0000","last_became_active":"2024-03-11T21:54:54.450311+0000","last_became_peered":"2024-03-11T21:54:54.450311+0000","last_unstale":"2024-03-11T21:55:32.066825+0000","last_undegraded":"2024-03-11T21:55:32.066825+0000","last_fullsized":"2024-03-11T21:55:32.066825+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:37:12.252483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946353+0000","last_change":"2024-03-11T21:54:50.414818+0000","last_active":"2024-03-11T21:55:31.946353+0000","last_peered":"2024-03-11T21:55:31.946353+0000","last_clean":"2024-03-11T21:55:31.946353+0000","last_became_active":"2024-03-11T21:54:50.414529+0000","last_became_peered":"2024-03-11T21:54:50.414529+0000","last_unstale":"2024-03-11T21:55:31.946353+0000","last_undegraded":"2024-03-11T21:55:31.946353+0000","last_fullsized":"2024-03-11T21:55:31.946353+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:22:43.069361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066450+0000","last_change":"2024-03-11T21:54:52.433312+0000","last_active":"2024-03-11T21:55:32.066450+0000","last_peered":"2024-03-11T21:55:32.066450+0000","last_clean":"2024-03-11T21:55:32.066450+0000","last_became_active":"2024-03-11T21:54:52.432982+0000","last_became_peered":"2024-03-11T21:54:52.432982+0000","last_unstale":"2024-03-11T21:55:32.066450+0000","last_undegraded":"2024-03-11T21:55:32.066450+0000","last_fullsized":"2024-03-11T21:55:32.066450+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:11:02.802808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946716+0000","last_change":"2024-03-11T21:54:56.538240+0000","last_active":"2024-03-11T21:55:31.946716+0000","last_peered":"2024-03-11T21:55:31.946716+0000","last_clean":"2024-03-11T21:55:31.946716+0000","last_became_active":"2024-03-11T21:54:56.537946+0000","last_became_peered":"2024-03-11T21:54:56.537946+0000","last_unstale":"2024-03-11T21:55:31.946716+0000","last_undegraded":"2024-03-11T21:55:31.946716+0000","last_fullsized":"2024-03-11T21:55:31.946716+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:08:37.097383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934253+0000","last_change":"2024-03-11T21:54:54.453321+0000","last_active":"2024-03-11T21:55:31.934253+0000","last_peered":"2024-03-11T21:55:31.934253+0000","last_clean":"2024-03-11T21:55:31.934253+0000","last_became_active":"2024-03-11T21:54:54.453006+0000","last_became_peered":"2024-03-11T21:54:54.453006+0000","last_unstale":"2024-03-11T21:55:31.934253+0000","last_undegraded":"2024-03-11T21:55:31.934253+0000","last_fullsized":"2024-03-11T21:55:31.934253+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:03:18.399494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.979847+0000","last_change":"2024-03-11T21:54:50.418065+0000","last_active":"2024-03-11T21:55:31.979847+0000","last_peered":"2024-03-11T21:55:31.979847+0000","last_clean":"2024-03-11T21:55:31.979847+0000","last_became_active":"2024-03-11T21:54:50.417634+0000","last_became_peered":"2024-03-11T21:54:50.417634+0000","last_unstale":"2024-03-11T21:55:31.979847+0000","last_undegraded":"2024-03-11T21:55:31.979847+0000","last_fullsized":"2024-03-11T21:55:31.979847+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:28.038598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":38,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066855+0000","last_change":"2024-03-11T21:54:52.434550+0000","last_active":"2024-03-11T21:55:32.066855+0000","last_peered":"2024-03-11T21:55:32.066855+0000","last_clean":"2024-03-11T21:55:32.066855+0000","last_became_active":"2024-03-11T21:54:52.434240+0000","last_became_peered":"2024-03-11T21:54:52.434240+0000","last_unstale":"2024-03-11T21:55:32.066855+0000","last_undegraded":"2024-03-11T21:55:32.066855+0000","last_fullsized":"2024-03-11T21:55:32.066855+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:38:08.562054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933959+0000","last_change":"2024-03-11T21:54:56.535781+0000","last_active":"2024-03-11T21:55:31.933959+0000","last_peered":"2024-03-11T21:55:31.933959+0000","last_clean":"2024-03-11T21:55:31.933959+0000","last_became_active":"2024-03-11T21:54:56.535484+0000","last_became_peered":"2024-03-11T21:54:56.535484+0000","last_unstale":"2024-03-11T21:55:31.933959+0000","last_undegraded":"2024-03-11T21:55:31.933959+0000","last_fullsized":"2024-03-11T21:55:31.933959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:23.390563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.947125+0000","last_change":"2024-03-11T21:54:50.420247+0000","last_active":"2024-03-11T21:55:31.947125+0000","last_peered":"2024-03-11T21:55:31.947125+0000","last_clean":"2024-03-11T21:55:31.947125+0000","last_became_active":"2024-03-11T21:54:50.419950+0000","last_became_peered":"2024-03-11T21:54:50.419950+0000","last_unstale":"2024-03-11T21:55:31.947125+0000","last_undegraded":"2024-03-11T21:55:31.947125+0000","last_fullsized":"2024-03-11T21:55:31.947125+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:59:27.644018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933690+0000","last_change":"2024-03-11T21:54:52.430702+0000","last_active":"2024-03-11T21:55:31.933690+0000","last_peered":"2024-03-11T21:55:31.933690+0000","last_clean":"2024-03-11T21:55:31.933690+0000","last_became_active":"2024-03-11T21:54:52.430384+0000","last_became_peered":"2024-03-11T21:54:52.430384+0000","last_unstale":"2024-03-11T21:55:31.933690+0000","last_undegraded":"2024-03-11T21:55:31.933690+0000","last_fullsized":"2024-03-11T21:55:31.933690+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:44:08.784012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472151+0000","last_change":"2024-03-11T21:54:54.455583+0000","last_active":"2024-03-11T21:56:05.472151+0000","last_peered":"2024-03-11T21:56:05.472151+0000","last_clean":"2024-03-11T21:56:05.472151+0000","last_became_active":"2024-03-11T21:54:54.455284+0000","last_became_peered":"2024-03-11T21:54:54.455284+0000","last_unstale":"2024-03-11T21:56:05.472151+0000","last_undegraded":"2024-03-11T21:56:05.472151+0000","last_fullsized":"2024-03-11T21:56:05.472151+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:38:26.161240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065609+0000","last_change":"2024-03-11T21:54:56.461870+0000","last_active":"2024-03-11T21:55:32.065609+0000","last_peered":"2024-03-11T21:55:32.065609+0000","last_clean":"2024-03-11T21:55:32.065609+0000","last_became_active":"2024-03-11T21:54:56.460346+0000","last_became_peered":"2024-03-11T21:54:56.460346+0000","last_unstale":"2024-03-11T21:55:32.065609+0000","last_undegraded":"2024-03-11T21:55:32.065609+0000","last_fullsized":"2024-03-11T21:55:32.065609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:06:43.135842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.947030+0000","last_change":"2024-03-11T21:54:50.415553+0000","last_active":"2024-03-11T21:55:31.947030+0000","last_peered":"2024-03-11T21:55:31.947030+0000","last_clean":"2024-03-11T21:55:31.947030+0000","last_became_active":"2024-03-11T21:54:50.414908+0000","last_became_peered":"2024-03-11T21:54:50.414908+0000","last_unstale":"2024-03-11T21:55:31.947030+0000","last_undegraded":"2024-03-11T21:55:31.947030+0000","last_fullsized":"2024-03-11T21:55:31.947030+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:54.526575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831808+0000","last_change":"2024-03-11T21:54:52.437879+0000","last_active":"2024-03-11T21:55:31.831808+0000","last_peered":"2024-03-11T21:55:31.831808+0000","last_clean":"2024-03-11T21:55:31.831808+0000","last_became_active":"2024-03-11T21:54:52.437569+0000","last_became_peered":"2024-03-11T21:54:52.437569+0000","last_unstale":"2024-03-11T21:55:31.831808+0000","last_undegraded":"2024-03-11T21:55:31.831808+0000","last_fullsized":"2024-03-11T21:55:31.831808+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:31.980225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832232+0000","last_change":"2024-03-11T21:54:54.448595+0000","last_active":"2024-03-11T21:55:31.832232+0000","last_peered":"2024-03-11T21:55:31.832232+0000","last_clean":"2024-03-11T21:55:31.832232+0000","last_became_active":"2024-03-11T21:54:54.443571+0000","last_became_peered":"2024-03-11T21:54:54.443571+0000","last_unstale":"2024-03-11T21:55:31.832232+0000","last_undegraded":"2024-03-11T21:55:31.832232+0000","last_fullsized":"2024-03-11T21:55:31.832232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:44.205274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946214+0000","last_change":"2024-03-11T21:54:56.460951+0000","last_active":"2024-03-11T21:55:31.946214+0000","last_peered":"2024-03-11T21:55:31.946214+0000","last_clean":"2024-03-11T21:55:31.946214+0000","last_became_active":"2024-03-11T21:54:56.460663+0000","last_became_peered":"2024-03-11T21:54:56.460663+0000","last_unstale":"2024-03-11T21:55:31.946214+0000","last_undegraded":"2024-03-11T21:55:31.946214+0000","last_fullsized":"2024-03-11T21:55:31.946214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:05:28.165394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832582+0000","last_change":"2024-03-11T21:54:54.448661+0000","last_active":"2024-03-11T21:55:31.832582+0000","last_peered":"2024-03-11T21:55:31.832582+0000","last_clean":"2024-03-11T21:55:31.832582+0000","last_became_active":"2024-03-11T21:54:54.448220+0000","last_became_peered":"2024-03-11T21:54:54.448220+0000","last_unstale":"2024-03-11T21:55:31.832582+0000","last_undegraded":"2024-03-11T21:55:31.832582+0000","last_fullsized":"2024-03-11T21:55:31.832582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:14:20.312541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946873+0000","last_change":"2024-03-11T21:54:50.418687+0000","last_active":"2024-03-11T21:55:31.946873+0000","last_peered":"2024-03-11T21:55:31.946873+0000","last_clean":"2024-03-11T21:55:31.946873+0000","last_became_active":"2024-03-11T21:54:50.418312+0000","last_became_peered":"2024-03-11T21:54:50.418312+0000","last_unstale":"2024-03-11T21:55:31.946873+0000","last_undegraded":"2024-03-11T21:55:31.946873+0000","last_fullsized":"2024-03-11T21:55:31.946873+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:45.376240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.980194+0000","last_change":"2024-03-11T21:54:52.430399+0000","last_active":"2024-03-11T21:55:31.980194+0000","last_peered":"2024-03-11T21:55:31.980194+0000","last_clean":"2024-03-11T21:55:31.980194+0000","last_became_active":"2024-03-11T21:54:52.430095+0000","last_became_peered":"2024-03-11T21:54:52.430095+0000","last_unstale":"2024-03-11T21:55:31.980194+0000","last_undegraded":"2024-03-11T21:55:31.980194+0000","last_fullsized":"2024-03-11T21:55:31.980194+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:12:40.577667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065124+0000","last_change":"2024-03-11T21:54:56.457418+0000","last_active":"2024-03-11T21:55:32.065124+0000","last_peered":"2024-03-11T21:55:32.065124+0000","last_clean":"2024-03-11T21:55:32.065124+0000","last_became_active":"2024-03-11T21:54:56.457005+0000","last_became_peered":"2024-03-11T21:54:56.457005+0000","last_unstale":"2024-03-11T21:55:32.065124+0000","last_undegraded":"2024-03-11T21:55:32.065124+0000","last_fullsized":"2024-03-11T21:55:32.065124+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:15:02.771265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832447+0000","last_change":"2024-03-11T21:54:54.446607+0000","last_active":"2024-03-11T21:55:31.832447+0000","last_peered":"2024-03-11T21:55:31.832447+0000","last_clean":"2024-03-11T21:55:31.832447+0000","last_became_active":"2024-03-11T21:54:54.446273+0000","last_became_peered":"2024-03-11T21:54:54.446273+0000","last_unstale":"2024-03-11T21:55:31.832447+0000","last_undegraded":"2024-03-11T21:55:31.832447+0000","last_fullsized":"2024-03-11T21:55:31.832447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:34:35.240809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934289+0000","last_change":"2024-03-11T21:54:50.416492+0000","last_active":"2024-03-11T21:55:31.934289+0000","last_peered":"2024-03-11T21:55:31.934289+0000","last_clean":"2024-03-11T21:55:31.934289+0000","last_became_active":"2024-03-11T21:54:50.416197+0000","last_became_peered":"2024-03-11T21:54:50.416197+0000","last_unstale":"2024-03-11T21:55:31.934289+0000","last_undegraded":"2024-03-11T21:55:31.934289+0000","last_fullsized":"2024-03-11T21:55:31.934289+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:07:12.077530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066666+0000","last_change":"2024-03-11T21:54:52.431538+0000","last_active":"2024-03-11T21:55:32.066666+0000","last_peered":"2024-03-11T21:55:32.066666+0000","last_clean":"2024-03-11T21:55:32.066666+0000","last_became_active":"2024-03-11T21:54:52.431141+0000","last_became_peered":"2024-03-11T21:54:52.431141+0000","last_unstale":"2024-03-11T21:55:32.066666+0000","last_undegraded":"2024-03-11T21:55:32.066666+0000","last_fullsized":"2024-03-11T21:55:32.066666+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:32:12.661431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066689+0000","last_change":"2024-03-11T21:54:56.460842+0000","last_active":"2024-03-11T21:55:32.066689+0000","last_peered":"2024-03-11T21:55:32.066689+0000","last_clean":"2024-03-11T21:55:32.066689+0000","last_became_active":"2024-03-11T21:54:56.460512+0000","last_became_peered":"2024-03-11T21:54:56.460512+0000","last_unstale":"2024-03-11T21:55:32.066689+0000","last_undegraded":"2024-03-11T21:55:32.066689+0000","last_fullsized":"2024-03-11T21:55:32.066689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:28:48.717744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.933800+0000","last_change":"2024-03-11T21:54:54.443087+0000","last_active":"2024-03-11T21:55:31.933800+0000","last_peered":"2024-03-11T21:55:31.933800+0000","last_clean":"2024-03-11T21:55:31.933800+0000","last_became_active":"2024-03-11T21:54:50.413769+0000","last_became_peered":"2024-03-11T21:54:50.413769+0000","last_unstale":"2024-03-11T21:55:31.933800+0000","last_undegraded":"2024-03-11T21:55:31.933800+0000","last_fullsized":"2024-03-11T21:55:31.933800+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:18.090345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020997749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831229+0000","last_change":"2024-03-11T21:54:50.417454+0000","last_active":"2024-03-11T21:55:31.831229+0000","last_peered":"2024-03-11T21:55:31.831229+0000","last_clean":"2024-03-11T21:55:31.831229+0000","last_became_active":"2024-03-11T21:54:50.417085+0000","last_became_peered":"2024-03-11T21:54:50.417085+0000","last_unstale":"2024-03-11T21:55:31.831229+0000","last_undegraded":"2024-03-11T21:55:31.831229+0000","last_fullsized":"2024-03-11T21:55:31.831229+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:20:17.362116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"71'29","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472152+0000","last_change":"2024-03-11T21:55:13.523981+0000","last_active":"2024-03-11T21:56:05.472152+0000","last_peered":"2024-03-11T21:56:05.472152+0000","last_clean":"2024-03-11T21:56:05.472152+0000","last_became_active":"2024-03-11T21:55:13.509349+0000","last_became_peered":"2024-03-11T21:55:13.509349+0000","last_unstale":"2024-03-11T21:56:05.472152+0000","last_undegraded":"2024-03-11T21:56:05.472152+0000","last_fullsized":"2024-03-11T21:56:05.472152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:12:14.846827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065665+0000","last_change":"2024-03-11T21:54:54.449510+0000","last_active":"2024-03-11T21:55:32.065665+0000","last_peered":"2024-03-11T21:55:32.065665+0000","last_clean":"2024-03-11T21:55:32.065665+0000","last_became_active":"2024-03-11T21:54:54.449098+0000","last_became_peered":"2024-03-11T21:54:54.449098+0000","last_unstale":"2024-03-11T21:55:32.065665+0000","last_undegraded":"2024-03-11T21:55:32.065665+0000","last_fullsized":"2024-03-11T21:55:32.065665+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:06:38.828874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946961+0000","last_change":"2024-03-11T21:54:56.535374+0000","last_active":"2024-03-11T21:55:31.946961+0000","last_peered":"2024-03-11T21:55:31.946961+0000","last_clean":"2024-03-11T21:55:31.946961+0000","last_became_active":"2024-03-11T21:54:56.535062+0000","last_became_peered":"2024-03-11T21:54:56.535062+0000","last_unstale":"2024-03-11T21:55:31.946961+0000","last_undegraded":"2024-03-11T21:55:31.946961+0000","last_fullsized":"2024-03-11T21:55:31.946961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:52:35.444035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830967+0000","last_change":"2024-03-11T21:54:50.415222+0000","last_active":"2024-03-11T21:55:31.830967+0000","last_peered":"2024-03-11T21:55:31.830967+0000","last_clean":"2024-03-11T21:55:31.830967+0000","last_became_active":"2024-03-11T21:54:50.414931+0000","last_became_peered":"2024-03-11T21:54:50.414931+0000","last_unstale":"2024-03-11T21:55:31.830967+0000","last_undegraded":"2024-03-11T21:55:31.830967+0000","last_fullsized":"2024-03-11T21:55:31.830967+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:43:28.948730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832471+0000","last_change":"2024-03-11T21:54:54.436020+0000","last_active":"2024-03-11T21:55:31.832471+0000","last_peered":"2024-03-11T21:55:31.832471+0000","last_clean":"2024-03-11T21:55:31.832471+0000","last_became_active":"2024-03-11T21:54:50.415862+0000","last_became_peered":"2024-03-11T21:54:50.415862+0000","last_unstale":"2024-03-11T21:55:31.832471+0000","last_undegraded":"2024-03-11T21:55:31.832471+0000","last_fullsized":"2024-03-11T21:55:31.832471+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:42:38.179547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013831500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832995+0000","last_change":"2024-03-11T21:55:13.516801+0000","last_active":"2024-03-11T21:55:31.832995+0000","last_peered":"2024-03-11T21:55:31.832995+0000","last_clean":"2024-03-11T21:55:31.832995+0000","last_became_active":"2024-03-11T21:55:13.508331+0000","last_became_peered":"2024-03-11T21:55:13.508331+0000","last_unstale":"2024-03-11T21:55:31.832995+0000","last_undegraded":"2024-03-11T21:55:31.832995+0000","last_fullsized":"2024-03-11T21:55:31.832995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:00.214523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832774+0000","last_change":"2024-03-11T21:54:54.449009+0000","last_active":"2024-03-11T21:55:31.832774+0000","last_peered":"2024-03-11T21:55:31.832774+0000","last_clean":"2024-03-11T21:55:31.832774+0000","last_became_active":"2024-03-11T21:54:54.448646+0000","last_became_peered":"2024-03-11T21:54:54.448646+0000","last_unstale":"2024-03-11T21:55:31.832774+0000","last_undegraded":"2024-03-11T21:55:31.832774+0000","last_fullsized":"2024-03-11T21:55:31.832774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:21:16.719560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934325+0000","last_change":"2024-03-11T21:54:56.461733+0000","last_active":"2024-03-11T21:55:31.934325+0000","last_peered":"2024-03-11T21:55:31.934325+0000","last_clean":"2024-03-11T21:55:31.934325+0000","last_became_active":"2024-03-11T21:54:56.461429+0000","last_became_peered":"2024-03-11T21:54:56.461429+0000","last_unstale":"2024-03-11T21:55:31.934325+0000","last_undegraded":"2024-03-11T21:55:31.934325+0000","last_fullsized":"2024-03-11T21:55:31.934325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:25.346326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'188","reported_seq":345,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:33.928751+0000","last_change":"2024-03-11T21:54:40.647817+0000","last_active":"2024-03-11T21:55:33.928751+0000","last_peered":"2024-03-11T21:55:33.928751+0000","last_clean":"2024-03-11T21:55:33.928751+0000","last_became_active":"2024-03-11T21:54:40.647391+0000","last_became_peered":"2024-03-11T21:54:40.647391+0000","last_unstale":"2024-03-11T21:55:33.928751+0000","last_undegraded":"2024-03-11T21:55:33.928751+0000","last_fullsized":"2024-03-11T21:55:33.928751+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:52:26.665210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:52:26.665210+0000","last_clean_scrub_stamp":"2024-03-11T21:52:26.665210+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:43:36.382773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065564+0000","last_change":"2024-03-11T21:54:50.421030+0000","last_active":"2024-03-11T21:55:32.065564+0000","last_peered":"2024-03-11T21:55:32.065564+0000","last_clean":"2024-03-11T21:55:32.065564+0000","last_became_active":"2024-03-11T21:54:50.420706+0000","last_became_peered":"2024-03-11T21:54:50.420706+0000","last_unstale":"2024-03-11T21:55:32.065564+0000","last_undegraded":"2024-03-11T21:55:32.065564+0000","last_fullsized":"2024-03-11T21:55:32.065564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:57:25.226971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472184+0000","last_change":"2024-03-11T21:54:52.433823+0000","last_active":"2024-03-11T21:56:05.472184+0000","last_peered":"2024-03-11T21:56:05.472184+0000","last_clean":"2024-03-11T21:56:05.472184+0000","last_became_active":"2024-03-11T21:54:52.432703+0000","last_became_peered":"2024-03-11T21:54:52.432703+0000","last_unstale":"2024-03-11T21:56:05.472184+0000","last_undegraded":"2024-03-11T21:56:05.472184+0000","last_fullsized":"2024-03-11T21:56:05.472184+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:02.400690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946696+0000","last_change":"2024-03-11T21:54:54.457179+0000","last_active":"2024-03-11T21:55:31.946696+0000","last_peered":"2024-03-11T21:55:31.946696+0000","last_clean":"2024-03-11T21:55:31.946696+0000","last_became_active":"2024-03-11T21:54:54.456871+0000","last_became_peered":"2024-03-11T21:54:54.456871+0000","last_unstale":"2024-03-11T21:55:31.946696+0000","last_undegraded":"2024-03-11T21:55:31.946696+0000","last_fullsized":"2024-03-11T21:55:31.946696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:37.721150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831238+0000","last_change":"2024-03-11T21:54:56.461667+0000","last_active":"2024-03-11T21:55:31.831238+0000","last_peered":"2024-03-11T21:55:31.831238+0000","last_clean":"2024-03-11T21:55:31.831238+0000","last_became_active":"2024-03-11T21:54:56.461406+0000","last_became_peered":"2024-03-11T21:54:56.461406+0000","last_unstale":"2024-03-11T21:55:31.831238+0000","last_undegraded":"2024-03-11T21:55:31.831238+0000","last_fullsized":"2024-03-11T21:55:31.831238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:36:21.434251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065457+0000","last_change":"2024-03-11T21:54:50.419579+0000","last_active":"2024-03-11T21:55:32.065457+0000","last_peered":"2024-03-11T21:55:32.065457+0000","last_clean":"2024-03-11T21:55:32.065457+0000","last_became_active":"2024-03-11T21:54:50.418728+0000","last_became_peered":"2024-03-11T21:54:50.418728+0000","last_unstale":"2024-03-11T21:55:32.065457+0000","last_undegraded":"2024-03-11T21:55:32.065457+0000","last_fullsized":"2024-03-11T21:55:32.065457+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:37:43.186317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":119,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:11.506549+0000","last_change":"2024-03-11T21:54:54.443746+0000","last_active":"2024-03-11T21:56:11.506549+0000","last_peered":"2024-03-11T21:56:11.506549+0000","last_clean":"2024-03-11T21:56:11.506549+0000","last_became_active":"2024-03-11T21:54:50.416729+0000","last_became_peered":"2024-03-11T21:54:50.416729+0000","last_unstale":"2024-03-11T21:56:11.506549+0000","last_undegraded":"2024-03-11T21:56:11.506549+0000","last_fullsized":"2024-03-11T21:56:11.506549+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:55:33.852709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043829300000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066033+0000","last_change":"2024-03-11T21:54:52.434522+0000","last_active":"2024-03-11T21:55:32.066033+0000","last_peered":"2024-03-11T21:55:32.066033+0000","last_clean":"2024-03-11T21:55:32.066033+0000","last_became_active":"2024-03-11T21:54:52.434192+0000","last_became_peered":"2024-03-11T21:54:52.434192+0000","last_unstale":"2024-03-11T21:55:32.066033+0000","last_undegraded":"2024-03-11T21:55:32.066033+0000","last_fullsized":"2024-03-11T21:55:32.066033+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:50.551618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934531+0000","last_change":"2024-03-11T21:54:54.450785+0000","last_active":"2024-03-11T21:55:31.934531+0000","last_peered":"2024-03-11T21:55:31.934531+0000","last_clean":"2024-03-11T21:55:31.934531+0000","last_became_active":"2024-03-11T21:54:54.450478+0000","last_became_peered":"2024-03-11T21:54:54.450478+0000","last_unstale":"2024-03-11T21:55:31.934531+0000","last_undegraded":"2024-03-11T21:55:31.934531+0000","last_fullsized":"2024-03-11T21:55:31.934531+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:45:17.941514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830654+0000","last_change":"2024-03-11T21:54:56.535560+0000","last_active":"2024-03-11T21:55:31.830654+0000","last_peered":"2024-03-11T21:55:31.830654+0000","last_clean":"2024-03-11T21:55:31.830654+0000","last_became_active":"2024-03-11T21:54:56.535161+0000","last_became_peered":"2024-03-11T21:54:56.535161+0000","last_unstale":"2024-03-11T21:55:31.830654+0000","last_undegraded":"2024-03-11T21:55:31.830654+0000","last_fullsized":"2024-03-11T21:55:31.830654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:00:27.195839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831383+0000","last_change":"2024-03-11T21:54:50.413755+0000","last_active":"2024-03-11T21:55:31.831383+0000","last_peered":"2024-03-11T21:55:31.831383+0000","last_clean":"2024-03-11T21:55:31.831383+0000","last_became_active":"2024-03-11T21:54:50.413453+0000","last_became_peered":"2024-03-11T21:54:50.413453+0000","last_unstale":"2024-03-11T21:55:31.831383+0000","last_undegraded":"2024-03-11T21:55:31.831383+0000","last_fullsized":"2024-03-11T21:55:31.831383+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:31:41.724551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":65,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066411+0000","last_change":"2024-03-11T21:54:52.430361+0000","last_active":"2024-03-11T21:55:32.066411+0000","last_peered":"2024-03-11T21:55:32.066411+0000","last_clean":"2024-03-11T21:55:32.066411+0000","last_became_active":"2024-03-11T21:54:52.429990+0000","last_became_peered":"2024-03-11T21:54:52.429990+0000","last_unstale":"2024-03-11T21:55:32.066411+0000","last_undegraded":"2024-03-11T21:55:32.066411+0000","last_fullsized":"2024-03-11T21:55:32.066411+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:56:30.808133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.980248+0000","last_change":"2024-03-11T21:54:54.451141+0000","last_active":"2024-03-11T21:55:31.980248+0000","last_peered":"2024-03-11T21:55:31.980248+0000","last_clean":"2024-03-11T21:55:31.980248+0000","last_became_active":"2024-03-11T21:54:54.450842+0000","last_became_peered":"2024-03-11T21:54:54.450842+0000","last_unstale":"2024-03-11T21:55:31.980248+0000","last_undegraded":"2024-03-11T21:55:31.980248+0000","last_fullsized":"2024-03-11T21:55:31.980248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:02.413080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064964+0000","last_change":"2024-03-11T21:54:56.461045+0000","last_active":"2024-03-11T21:55:32.064964+0000","last_peered":"2024-03-11T21:55:32.064964+0000","last_clean":"2024-03-11T21:55:32.064964+0000","last_became_active":"2024-03-11T21:54:56.460761+0000","last_became_peered":"2024-03-11T21:54:56.460761+0000","last_unstale":"2024-03-11T21:55:32.064964+0000","last_undegraded":"2024-03-11T21:55:32.064964+0000","last_fullsized":"2024-03-11T21:55:32.064964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:40:53.333395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934263+0000","last_change":"2024-03-11T21:54:50.415035+0000","last_active":"2024-03-11T21:55:31.934263+0000","last_peered":"2024-03-11T21:55:31.934263+0000","last_clean":"2024-03-11T21:55:31.934263+0000","last_became_active":"2024-03-11T21:54:50.413522+0000","last_became_peered":"2024-03-11T21:54:50.413522+0000","last_unstale":"2024-03-11T21:55:31.934263+0000","last_undegraded":"2024-03-11T21:55:31.934263+0000","last_fullsized":"2024-03-11T21:55:31.934263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:58.564393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831128+0000","last_change":"2024-03-11T21:54:52.429688+0000","last_active":"2024-03-11T21:55:31.831128+0000","last_peered":"2024-03-11T21:55:31.831128+0000","last_clean":"2024-03-11T21:55:31.831128+0000","last_became_active":"2024-03-11T21:54:52.429356+0000","last_became_peered":"2024-03-11T21:54:52.429356+0000","last_unstale":"2024-03-11T21:55:31.831128+0000","last_undegraded":"2024-03-11T21:55:31.831128+0000","last_fullsized":"2024-03-11T21:55:31.831128+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:49:24.721134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.471946+0000","last_change":"2024-03-11T21:54:54.450815+0000","last_active":"2024-03-11T21:56:05.471946+0000","last_peered":"2024-03-11T21:56:05.471946+0000","last_clean":"2024-03-11T21:56:05.471946+0000","last_became_active":"2024-03-11T21:54:54.450407+0000","last_became_peered":"2024-03-11T21:54:54.450407+0000","last_unstale":"2024-03-11T21:56:05.471946+0000","last_undegraded":"2024-03-11T21:56:05.471946+0000","last_fullsized":"2024-03-11T21:56:05.471946+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:53:27.167368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.946609+0000","last_change":"2024-03-11T21:54:56.536130+0000","last_active":"2024-03-11T21:55:31.946609+0000","last_peered":"2024-03-11T21:55:31.946609+0000","last_clean":"2024-03-11T21:55:31.946609+0000","last_became_active":"2024-03-11T21:54:56.535833+0000","last_became_peered":"2024-03-11T21:54:56.535833+0000","last_unstale":"2024-03-11T21:55:31.946609+0000","last_undegraded":"2024-03-11T21:55:31.946609+0000","last_fullsized":"2024-03-11T21:55:31.946609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:54:36.263864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066280+0000","last_change":"2024-03-11T21:54:50.417027+0000","last_active":"2024-03-11T21:55:32.066280+0000","last_peered":"2024-03-11T21:55:32.066280+0000","last_clean":"2024-03-11T21:55:32.066280+0000","last_became_active":"2024-03-11T21:54:50.416723+0000","last_became_peered":"2024-03-11T21:54:50.416723+0000","last_unstale":"2024-03-11T21:55:32.066280+0000","last_undegraded":"2024-03-11T21:55:32.066280+0000","last_fullsized":"2024-03-11T21:55:32.066280+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:17:12.360593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831125+0000","last_change":"2024-03-11T21:54:52.431466+0000","last_active":"2024-03-11T21:55:31.831125+0000","last_peered":"2024-03-11T21:55:31.831125+0000","last_clean":"2024-03-11T21:55:31.831125+0000","last_became_active":"2024-03-11T21:54:52.431151+0000","last_became_peered":"2024-03-11T21:54:52.431151+0000","last_unstale":"2024-03-11T21:55:31.831125+0000","last_undegraded":"2024-03-11T21:55:31.831125+0000","last_fullsized":"2024-03-11T21:55:31.831125+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:08.938448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.471972+0000","last_change":"2024-03-11T21:54:54.445374+0000","last_active":"2024-03-11T21:56:05.471972+0000","last_peered":"2024-03-11T21:56:05.471972+0000","last_clean":"2024-03-11T21:56:05.471972+0000","last_became_active":"2024-03-11T21:54:54.445070+0000","last_became_peered":"2024-03-11T21:54:54.445070+0000","last_unstale":"2024-03-11T21:56:05.471972+0000","last_undegraded":"2024-03-11T21:56:05.471972+0000","last_fullsized":"2024-03-11T21:56:05.471972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:38.796149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831972+0000","last_change":"2024-03-11T21:54:56.452272+0000","last_active":"2024-03-11T21:55:31.831972+0000","last_peered":"2024-03-11T21:55:31.831972+0000","last_clean":"2024-03-11T21:55:31.831972+0000","last_became_active":"2024-03-11T21:54:56.452077+0000","last_became_peered":"2024-03-11T21:54:56.452077+0000","last_unstale":"2024-03-11T21:55:31.831972+0000","last_undegraded":"2024-03-11T21:55:31.831972+0000","last_fullsized":"2024-03-11T21:55:31.831972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:40.718941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.066769+0000","last_change":"2024-03-11T21:54:50.418599+0000","last_active":"2024-03-11T21:55:32.066769+0000","last_peered":"2024-03-11T21:55:32.066769+0000","last_clean":"2024-03-11T21:55:32.066769+0000","last_became_active":"2024-03-11T21:54:50.418290+0000","last_became_peered":"2024-03-11T21:54:50.418290+0000","last_unstale":"2024-03-11T21:55:32.066769+0000","last_undegraded":"2024-03-11T21:55:32.066769+0000","last_fullsized":"2024-03-11T21:55:32.066769+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:09:18.950583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065006+0000","last_change":"2024-03-11T21:54:52.436091+0000","last_active":"2024-03-11T21:55:32.065006+0000","last_peered":"2024-03-11T21:55:32.065006+0000","last_clean":"2024-03-11T21:55:32.065006+0000","last_became_active":"2024-03-11T21:54:52.433159+0000","last_became_peered":"2024-03-11T21:54:52.433159+0000","last_unstale":"2024-03-11T21:55:32.065006+0000","last_undegraded":"2024-03-11T21:55:32.065006+0000","last_fullsized":"2024-03-11T21:55:32.065006+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:17.575810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.830966+0000","last_change":"2024-03-11T21:54:54.454078+0000","last_active":"2024-03-11T21:55:31.830966+0000","last_peered":"2024-03-11T21:55:31.830966+0000","last_clean":"2024-03-11T21:55:31.830966+0000","last_became_active":"2024-03-11T21:54:54.453731+0000","last_became_peered":"2024-03-11T21:54:54.453731+0000","last_unstale":"2024-03-11T21:55:31.830966+0000","last_undegraded":"2024-03-11T21:55:31.830966+0000","last_fullsized":"2024-03-11T21:55:31.830966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:53:08.384585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934086+0000","last_change":"2024-03-11T21:54:56.456245+0000","last_active":"2024-03-11T21:55:31.934086+0000","last_peered":"2024-03-11T21:55:31.934086+0000","last_clean":"2024-03-11T21:55:31.934086+0000","last_became_active":"2024-03-11T21:54:56.455948+0000","last_became_peered":"2024-03-11T21:54:56.455948+0000","last_unstale":"2024-03-11T21:55:31.934086+0000","last_undegraded":"2024-03-11T21:55:31.934086+0000","last_fullsized":"2024-03-11T21:55:31.934086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:42:06.952202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.472174+0000","last_change":"2024-03-11T21:54:54.446693+0000","last_active":"2024-03-11T21:56:05.472174+0000","last_peered":"2024-03-11T21:56:05.472174+0000","last_clean":"2024-03-11T21:56:05.472174+0000","last_became_active":"2024-03-11T21:54:54.446397+0000","last_became_peered":"2024-03-11T21:54:54.446397+0000","last_unstale":"2024-03-11T21:56:05.472174+0000","last_undegraded":"2024-03-11T21:56:05.472174+0000","last_fullsized":"2024-03-11T21:56:05.472174+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:38.116626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.064952+0000","last_change":"2024-03-11T21:54:50.413463+0000","last_active":"2024-03-11T21:55:32.064952+0000","last_peered":"2024-03-11T21:55:32.064952+0000","last_clean":"2024-03-11T21:55:32.064952+0000","last_became_active":"2024-03-11T21:54:50.413167+0000","last_became_peered":"2024-03-11T21:54:50.413167+0000","last_unstale":"2024-03-11T21:55:32.064952+0000","last_undegraded":"2024-03-11T21:55:32.064952+0000","last_fullsized":"2024-03-11T21:55:32.064952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:58.191595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831802+0000","last_change":"2024-03-11T21:54:52.434870+0000","last_active":"2024-03-11T21:55:31.831802+0000","last_peered":"2024-03-11T21:55:31.831802+0000","last_clean":"2024-03-11T21:55:31.831802+0000","last_became_active":"2024-03-11T21:54:52.434127+0000","last_became_peered":"2024-03-11T21:54:52.434127+0000","last_unstale":"2024-03-11T21:55:31.831802+0000","last_undegraded":"2024-03-11T21:55:31.831802+0000","last_fullsized":"2024-03-11T21:55:31.831802+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:55:35.588050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065867+0000","last_change":"2024-03-11T21:54:56.535576+0000","last_active":"2024-03-11T21:55:32.065867+0000","last_peered":"2024-03-11T21:55:32.065867+0000","last_clean":"2024-03-11T21:55:32.065867+0000","last_became_active":"2024-03-11T21:54:56.534987+0000","last_became_peered":"2024-03-11T21:54:56.534987+0000","last_unstale":"2024-03-11T21:55:32.065867+0000","last_undegraded":"2024-03-11T21:55:32.065867+0000","last_fullsized":"2024-03-11T21:55:32.065867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:57.107172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.934499+0000","last_change":"2024-03-11T21:54:54.449099+0000","last_active":"2024-03-11T21:55:31.934499+0000","last_peered":"2024-03-11T21:55:31.934499+0000","last_clean":"2024-03-11T21:55:31.934499+0000","last_became_active":"2024-03-11T21:54:54.448782+0000","last_became_peered":"2024-03-11T21:54:54.448782+0000","last_unstale":"2024-03-11T21:55:31.934499+0000","last_undegraded":"2024-03-11T21:55:31.934499+0000","last_fullsized":"2024-03-11T21:55:31.934499+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:06:45.651830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831023+0000","last_change":"2024-03-11T21:54:50.415240+0000","last_active":"2024-03-11T21:55:31.831023+0000","last_peered":"2024-03-11T21:55:31.831023+0000","last_clean":"2024-03-11T21:55:31.831023+0000","last_became_active":"2024-03-11T21:54:50.414696+0000","last_became_peered":"2024-03-11T21:54:50.414696+0000","last_unstale":"2024-03-11T21:55:31.831023+0000","last_undegraded":"2024-03-11T21:55:31.831023+0000","last_fullsized":"2024-03-11T21:55:31.831023+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:11:24.610801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831332+0000","last_change":"2024-03-11T21:54:52.435284+0000","last_active":"2024-03-11T21:55:31.831332+0000","last_peered":"2024-03-11T21:55:31.831332+0000","last_clean":"2024-03-11T21:55:31.831332+0000","last_became_active":"2024-03-11T21:54:52.434854+0000","last_became_peered":"2024-03-11T21:54:52.434854+0000","last_unstale":"2024-03-11T21:55:31.831332+0000","last_undegraded":"2024-03-11T21:55:31.831332+0000","last_fullsized":"2024-03-11T21:55:31.831332+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:50:21.836802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.832877+0000","last_change":"2024-03-11T21:54:56.461228+0000","last_active":"2024-03-11T21:55:31.832877+0000","last_peered":"2024-03-11T21:55:31.832877+0000","last_clean":"2024-03-11T21:55:31.832877+0000","last_became_active":"2024-03-11T21:54:56.460838+0000","last_became_peered":"2024-03-11T21:54:56.460838+0000","last_unstale":"2024-03-11T21:55:31.832877+0000","last_undegraded":"2024-03-11T21:55:31.832877+0000","last_fullsized":"2024-03-11T21:55:31.832877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:00.935613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831556+0000","last_change":"2024-03-11T21:54:54.448062+0000","last_active":"2024-03-11T21:55:31.831556+0000","last_peered":"2024-03-11T21:55:31.831556+0000","last_clean":"2024-03-11T21:55:31.831556+0000","last_became_active":"2024-03-11T21:54:54.447705+0000","last_became_peered":"2024-03-11T21:54:54.447705+0000","last_unstale":"2024-03-11T21:55:31.831556+0000","last_undegraded":"2024-03-11T21:55:31.831556+0000","last_fullsized":"2024-03-11T21:55:31.831556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:43:47.716024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.979996+0000","last_change":"2024-03-11T21:54:50.416934+0000","last_active":"2024-03-11T21:55:31.979996+0000","last_peered":"2024-03-11T21:55:31.979996+0000","last_clean":"2024-03-11T21:55:31.979996+0000","last_became_active":"2024-03-11T21:54:50.416640+0000","last_became_peered":"2024-03-11T21:54:50.416640+0000","last_unstale":"2024-03-11T21:55:31.979996+0000","last_undegraded":"2024-03-11T21:55:31.979996+0000","last_fullsized":"2024-03-11T21:55:31.979996+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:00:51.800411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831845+0000","last_change":"2024-03-11T21:54:52.433430+0000","last_active":"2024-03-11T21:55:31.831845+0000","last_peered":"2024-03-11T21:55:31.831845+0000","last_clean":"2024-03-11T21:55:31.831845+0000","last_became_active":"2024-03-11T21:54:52.433140+0000","last_became_peered":"2024-03-11T21:54:52.433140+0000","last_unstale":"2024-03-11T21:55:31.831845+0000","last_undegraded":"2024-03-11T21:55:31.831845+0000","last_fullsized":"2024-03-11T21:55:31.831845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:25:57.587156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831664+0000","last_change":"2024-03-11T21:54:56.448523+0000","last_active":"2024-03-11T21:55:31.831664+0000","last_peered":"2024-03-11T21:55:31.831664+0000","last_clean":"2024-03-11T21:55:31.831664+0000","last_became_active":"2024-03-11T21:54:56.448190+0000","last_became_peered":"2024-03-11T21:54:56.448190+0000","last_unstale":"2024-03-11T21:55:31.831664+0000","last_undegraded":"2024-03-11T21:55:31.831664+0000","last_fullsized":"2024-03-11T21:55:31.831664+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:55.425738+0000","last_clean_scrub_stamp":"2024-03-11T21:54:55.425738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:06:12.068288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:56:05.471973+0000","last_change":"2024-03-11T21:54:54.448651+0000","last_active":"2024-03-11T21:56:05.471973+0000","last_peered":"2024-03-11T21:56:05.471973+0000","last_clean":"2024-03-11T21:56:05.471973+0000","last_became_active":"2024-03-11T21:54:54.448263+0000","last_became_peered":"2024-03-11T21:54:54.448263+0000","last_unstale":"2024-03-11T21:56:05.471973+0000","last_undegraded":"2024-03-11T21:56:05.471973+0000","last_fullsized":"2024-03-11T21:56:05.471973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:53.413886+0000","last_clean_scrub_stamp":"2024-03-11T21:54:53.413886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:18.422914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:32.065789+0000","last_change":"2024-03-11T21:54:50.419727+0000","last_active":"2024-03-11T21:55:32.065789+0000","last_peered":"2024-03-11T21:55:32.065789+0000","last_clean":"2024-03-11T21:55:32.065789+0000","last_became_active":"2024-03-11T21:54:50.419444+0000","last_became_peered":"2024-03-11T21:54:50.419444+0000","last_unstale":"2024-03-11T21:55:32.065789+0000","last_undegraded":"2024-03-11T21:55:32.065789+0000","last_fullsized":"2024-03-11T21:55:32.065789+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:49.386123+0000","last_clean_scrub_stamp":"2024-03-11T21:54:49.386123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:25.570221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T21:55:31.831573+0000","last_change":"2024-03-11T21:54:52.434062+0000","last_active":"2024-03-11T21:55:31.831573+0000","last_peered":"2024-03-11T21:55:31.831573+0000","last_clean":"2024-03-11T21:55:31.831573+0000","last_became_active":"2024-03-11T21:54:52.433752+0000","last_became_peered":"2024-03-11T21:54:52.433752+0000","last_unstale":"2024-03-11T21:55:31.831573+0000","last_undegraded":"2024-03-11T21:55:31.831573+0000","last_fullsized":"2024-03-11T21:55:31.831573+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:54:51.398449+0000","last_clean_scrub_stamp":"2024-03-11T21:54:51.398449+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:00:33.622518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201301,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28392,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712664,"statfs":{"total":95990841344,"available":95961767936,"internally_reserved":0,"allocated":1404928,"data_stored":926806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32532,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708524,"statfs":{"total":95990841344,"available":95957528576,"internally_reserved":0,"allocated":1384448,"data_stored":925755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713236,"statfs":{"total":95990841344,"available":95962353664,"internally_reserved":0,"allocated":819200,"data_stored":335885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28388,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712668,"statfs":{"total":95990841344,"available":95961772032,"internally_reserved":0,"allocated":1400832,"data_stored":925662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713256,"statfs":{"total":95990841344,"available":95962374144,"internally_reserved":0,"allocated":798720,"data_stored":335452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709072,"statfs":{"total":95990841344,"available":95958089728,"internally_reserved":0,"allocated":823296,"data_stored":335404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713240,"statfs":{"total":95990841344,"available":95962357760,"internally_reserved":0,"allocated":815104,"data_stored":335503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724576,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709088,"statfs":{"total":95990841344,"available":95958106112,"internally_reserved":0,"allocated":806912,"data_stored":336057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:56:20.375 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T21:56:20.375 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T21:56:20.376 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T21:56:20.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T21:56:20.623 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:20.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:20 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ::ffff:172.21.15.110 - - [11/Mar/2024:21:56:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:56:22.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:21 smithi102 ceph-mon[25338]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:22.116 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:21 smithi102 ceph-mon[34177]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:22.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:21 smithi110 ceph-mon[30535]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:22.600 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-11T21:56:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T21:56:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:22 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:22 smithi102 ceph-mon[25338]: from='client.? 172.21.15.102:0/1120558324' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:56:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:22 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:22 smithi102 ceph-mon[34177]: from='client.? 172.21.15.102:0/1120558324' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:56:23.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:22 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:23.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:22 smithi110 ceph-mon[30535]: from='client.? 172.21.15.102:0/1120558324' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:56:23.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:56:22 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:56:23.308 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T21:56:23.308 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T21:56:23.308 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T21:56:23.319 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-11T21:56:23.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-11T21:56:23.560 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:24.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:23 smithi102 ceph-mon[25338]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:24.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:23 smithi102 ceph-mon[25338]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:24.026 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:23 smithi102 ceph-mon[34177]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:24.026 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:23 smithi102 ceph-mon[34177]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:24.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:23 smithi110 ceph-mon[30535]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:24.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:23 smithi110 ceph-mon[30535]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:25.166 INFO:teuthology.orchestra.run.smithi102.stdout:167 167 2024-03-11T21:56:25.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T21:56:26.167 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:25 smithi102 ceph-mon[25338]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:25 smithi102 ceph-mon[34177]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:26.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:25 smithi110 ceph-mon[30535]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:27.665 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:56:27 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:56:27.491Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002391732s 2024-03-11T21:56:27.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:27 smithi102 ceph-mon[25338]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:27 smithi102 ceph-mon[34177]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:28.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:27 smithi110 ceph-mon[30535]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:28.485 INFO:teuthology.orchestra.run.smithi102.stdout:Backend: cephadm 2024-03-11T21:56:28.485 INFO:teuthology.orchestra.run.smithi102.stdout:Available: Yes 2024-03-11T21:56:28.485 INFO:teuthology.orchestra.run.smithi102.stdout:Paused: No 2024-03-11T21:56:29.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:28 smithi102 ceph-mon[25338]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:29.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:28 smithi102 ceph-mon[34177]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:29.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:28 smithi110 ceph-mon[30535]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:29.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T21:56:29.553 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:29.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:29 smithi102 ceph-mon[25338]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:29.997 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:29 smithi102 ceph-mon[34177]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:30.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:29 smithi110 ceph-mon[30535]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:30.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:30 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ::ffff:172.21.15.110 - - [11/Mar/2024:21:56:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:56:31.358 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 starting - - - - 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:iscsi.iscsi.a smithi110 *:5000 running (89s) 56s ago 89s - - 3.6 a6477a474d71 6e2c30803a34 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi110 *:8443,8765 running (5m) 56s ago 5m - - 19.0.0-1607-g94a53970 a6477a474d71 41c8f46117c1 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:9283,8765 running (7m) 32s ago 7m 480M - 19.0.0-1607-g94a53970 a6477a474d71 e2bcbbac623f 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (7m) 32s ago 7m 53.2M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e26e085d973f 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi110 running (5m) 56s ago 5m - 2048M 19.0.0-1607-g94a53970 a6477a474d71 6ebd0e2e1fe2 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (5m) 32s ago 5m 49.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d65f8f2d5f2e 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (5m) 32s ago 5m 49.2M 3760M 19.0.0-1607-g94a53970 a6477a474d71 7d7ec9520d00 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (4m) 32s ago 4m 47.3M 3760M 19.0.0-1607-g94a53970 a6477a474d71 1c55d0dd19a0 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (4m) 32s ago 4m 42.8M 3760M 19.0.0-1607-g94a53970 a6477a474d71 bc79c903fefb 2024-03-11T21:56:31.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (3m) 32s ago 3m 52.8M 3760M 19.0.0-1607-g94a53970 a6477a474d71 fee6d2682609 2024-03-11T21:56:31.360 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi110 running (3m) 56s ago 3m - 3760M 19.0.0-1607-g94a53970 a6477a474d71 b6a67b672a5b 2024-03-11T21:56:31.360 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi110 running (2m) 56s ago 2m - 3760M 19.0.0-1607-g94a53970 a6477a474d71 7b2c617469b3 2024-03-11T21:56:31.360 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi110 running (2m) 56s ago 2m - 3760M 19.0.0-1607-g94a53970 a6477a474d71 27dd92278eed 2024-03-11T21:56:31.360 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi110 running (2m) 56s ago 2m - 3760M 19.0.0-1607-g94a53970 a6477a474d71 9f922576ebac 2024-03-11T21:56:31.360 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi110 *:9095 running (68s) 56s ago 67s - - 2.43.0 a07b618ecd1d ae0fd8d4739e 2024-03-11T21:56:31.360 INFO:teuthology.orchestra.run.smithi102.stdout:rgw.foo.a smithi102 *:80 running (103s) 32s ago 102s 77.3M - 19.0.0-1607-g94a53970 a6477a474d71 7543b2522bb3 2024-03-11T21:56:32.067 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:31 smithi102 ceph-mon[25338]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:32.067 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:31 smithi102 ceph-mon[25338]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:32.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:31 smithi102 ceph-mon[34177]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:32.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:31 smithi102 ceph-mon[34177]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:32.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T21:56:32.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:31 smithi110 ceph-mon[30535]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:32.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:31 smithi110 ceph-mon[30535]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:32.334 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:32 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:33.150 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:32 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:33.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:32 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:33.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:56:32 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:56:34.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:33 smithi102 ceph-mon[25338]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:34.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:33 smithi102 ceph-mon[34177]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:34.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:33 smithi110 ceph-mon[30535]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager ?:9093,9094 0/1 - 85s smithi102=a;count:1 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:grafana ?:3000 0/1 - 81s smithi110=a;count:1 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:iscsi.datapool ?:5000 1/1 59s ago 97s smithi110=iscsi.a;count:1 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:mgr 2/2 59s ago 5m smithi102=y;smithi110=x;count:2 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:mon 3/3 59s ago 5m smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi110:172.21.15.110=b;count:3 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter ?:9100 0/2 - 89s smithi102=a;smithi110=b;count:2 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:osd 8 59s ago - 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus ?:9095 1/1 59s ago 93s smithi110=a;count:1 2024-03-11T21:56:34.253 INFO:teuthology.orchestra.run.smithi102.stdout:rgw.foo.a ?:80 1/1 35s ago 105s smithi102=foo.a;count:1 2024-03-11T21:56:34.993 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:34 smithi102 ceph-mon[25338]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:34.993 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:34 smithi102 ceph-mon[25338]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:34.994 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:34 smithi102 ceph-mon[34177]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:34.994 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:34 smithi102 ceph-mon[34177]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:35.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T21:56:35.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:34 smithi110 ceph-mon[30535]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:35.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:34 smithi110 ceph-mon[30535]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:35.260 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:36.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:36 smithi110 ceph-mon[30535]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:56:37.134 INFO:teuthology.orchestra.run.smithi102.stdout:HOST ADDR LABELS STATUS 2024-03-11T21:56:37.134 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 172.21.15.102 2024-03-11T21:56:37.134 INFO:teuthology.orchestra.run.smithi102.stdout:smithi110 172.21.15.110 2024-03-11T21:56:37.134 INFO:teuthology.orchestra.run.smithi102.stdout:2 hosts in cluster 2024-03-11T21:56:37.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:36 smithi102 ceph-mon[25338]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:56:37.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:36 smithi102 ceph-mon[34177]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:56:37.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T21:56:37.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:37 smithi110 ceph-mon[30535]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:37 smithi102 ceph-mon[25338]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:38.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:37 smithi102 ceph-mon[34177]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:38.466 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 systemd[1]: Starting Ceph grafana.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:56:38.739 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 podman[70090]: 2024-03-11 21:56:38.375004897 +0000 UTC m=+0.018038634 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-11T21:56:38.739 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 podman[70090]: 2024-03-11 21:56:38.505640147 +0000 UTC m=+0.148673924 container create b9cbc7756b8b7184ed6a8aa4bc11e5150bd26a3cbcf8d4856f555ba66ea644a9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a, maintainer=Paul Cuzner , name=ubi8, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, version=8.5, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, architecture=x86_64, io.openshift.expose-services=) 2024-03-11T21:56:38.740 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 podman[70090]: 2024-03-11 21:56:38.674428937 +0000 UTC m=+0.317462672 container init b9cbc7756b8b7184ed6a8aa4bc11e5150bd26a3cbcf8d4856f555ba66ea644a9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a, name=ubi8, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., version=8.5, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , architecture=x86_64, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T21:56:38.740 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 podman[70090]: 2024-03-11 21:56:38.676860149 +0000 UTC m=+0.319893900 container start b9cbc7756b8b7184ed6a8aa4bc11e5150bd26a3cbcf8d4856f555ba66ea644a9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, io.openshift.expose-services=, vcs-type=git, io.openshift.tags=base rhel8, distribution-scope=public) 2024-03-11T21:56:38.990 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 bash[70090]: b9cbc7756b8b7184ed6a8aa4bc11e5150bd26a3cbcf8d4856f555ba66ea644a9 2024-03-11T21:56:38.990 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 systemd[1]: Started Ceph grafana.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:56:38.990 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829267069Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T21:56:38.990 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829466032Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T21:56:38.990 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829484206Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T21:56:38.990 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829491942Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T21:56:38.990 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829497996Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T21:56:38.990 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829503904Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T21:56:38.990 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829509564Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829515312Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829521154Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829526773Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829532096Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829537406Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.829543046Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=settings t=2024-03-11T21:56:38.82954869Z level=info msg="App mode production" 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=sqlstore t=2024-03-11T21:56:38.82959372Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=sqlstore t=2024-03-11T21:56:38.829609375Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.830491129Z level=info msg="Starting DB migrations" 2024-03-11T21:56:38.991 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.83056704Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T21:56:38.992 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.831128406Z level=info msg="Executing migration" id="create user table" 2024-03-11T21:56:38.992 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.831579982Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T21:56:38.992 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.832038641Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T21:56:38.992 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.832470678Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T21:56:38.992 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.83303273Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T21:56:38.992 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.833470439Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T21:56:38.992 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.834183726Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.834557346Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.834939604Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.835369481Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.835660605Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.835985354Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.836384907Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.83661832Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.837020873Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.837351468Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.837712065Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.838101536Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.838493172Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.839590482Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T21:56:38.993 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.83999172Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.840392349Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.840770857Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.841174165Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.84157121Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.841837737Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.8421983Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.842522878Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.842847701Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.843200838Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.843795793Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T21:56:38.994 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.844165587Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.844508018Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.844843666Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.845206012Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.845600906Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.845939605Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.846252568Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.846525952Z level=info msg="Executing migration" id="create star table" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.846830458Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.847222174Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.847589777Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.847969748Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.848312045Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.848698098Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T21:56:38.995 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.849104572Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.849464525Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.84970631Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.849996377Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.850272256Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.850619949Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.851002153Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.851417663Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.851777096Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.852181506Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.85251662Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.853324364Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T21:56:38.996 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.853676222Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.854077339Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.854470814Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.854739544Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.855107687Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.855342105Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.855752185Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.856203185Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.856647417Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.857034797Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.85747542Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.857865969Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.858243978Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.85846955Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T21:56:38.998 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.858695627Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.85922224Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.859705431Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.860164963Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.860613239Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.860906057Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.861303999Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.861637645Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.861955788Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.862357396Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.862748306Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.863633466Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.86400512Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.864394667Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T21:56:38.999 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.864814042Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.865183046Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.865530147Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.866224018Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.866734521Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.867095245Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.86742106Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.867902949Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.868615357Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.869162509Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.869709299Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.870233736Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.870738068Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T21:56:39.000 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.871234846Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.872248405Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.872610079Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.8729831Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.873356773Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.873634678Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.874155443Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.874630833Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.874860041Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.875133235Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.875574087Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.875818759Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.876085991Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.876549475Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.876797682Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T21:56:39.001 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.877183977Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.877551684Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.877910642Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.878320165Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.878675409Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.879095953Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.879456584Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.87980921Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.880165379Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.881169Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.881520617Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.881868316Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.882268403Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.88265855Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T21:56:39.002 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.882978601Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.883289865Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.883501219Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.884003422Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.884526537Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.884766271Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.885302943Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.88576662Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.886129427Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.886420296Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.886736152Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.887192075Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.887550017Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.887933293Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T21:56:39.003 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.88818686Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.888404415Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.888920094Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.889426502Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.889676683Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.890035706Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.890411445Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.89063978Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.89101351Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.891454333Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.892084809Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.892361849Z level=info msg="Executing migration" id="create session table" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.892759213Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.893045368Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T21:56:39.004 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.893297787Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.893644508Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.894020404Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.894278378Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.894500255Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.894737185Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.895018343Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.895384463Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.895622333Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.8962549Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.896541157Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.897148647Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.897732565Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T21:56:39.005 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.898013742Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.898413669Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.898856736Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.899285882Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.899711757Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.90017322Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.900591029Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.900990943Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.901397975Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.901734576Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.903097343Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.903447816Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.903866525Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T21:56:39.006 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.904167046Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.904469851Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.904810402Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.90542072Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.90605503Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.906685396Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.907350853Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.907768219Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.908046581Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.908287479Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.908637025Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T21:56:39.007 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.909063816Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.909423389Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.909763038Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.910184689Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.910775921Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.911416046Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.9116724Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.912103965Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.912498741Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.91315657Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T21:56:39.009 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.913427658Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.913806582Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.914207181Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.914480339Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.914912542Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.915348213Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.91572532Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.916119041Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.916536355Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.91696207Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.917209162Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.917805213Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.918178183Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.918759309Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T21:56:39.010 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.919134892Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.919517992Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.919863143Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.921429692Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.921774889Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.92216303Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.922420609Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.922703055Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.922977601Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.923594372Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.924289864Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.924680186Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.925072238Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.925353469Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.925987358Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T21:56:39.011 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.926372891Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.926609995Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.926937954Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.927300583Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.927649492Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.928069045Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.928446964Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.928776718Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.929155182Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.92939594Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.929764394Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.930133857Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T21:56:39.012 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.930508975Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.930933925Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.931212035Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.931456755Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.931688046Z level=info msg="Executing migration" id="create team table" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.932046758Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.932447626Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.932829394Z level=info msg="Executing migration" id="create team member table" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.933176107Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.933571051Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.933967737Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.934364383Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.935238735Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.935939397Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.93661058Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T21:56:39.013 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.936991609Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.937425102Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.937892815Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.93829735Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.93873244Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.939120539Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.939501183Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.939892492Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.940266348Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.940513626Z level=info msg="Executing migration" id="create tag table" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.940857437Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.941234505Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.941571574Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T21:56:39.014 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.941956292Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.942319182Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.944108197Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.944458852Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.944851471Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.945133463Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.945420927Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.945727357Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.946120447Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.946372928Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.947071039Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.947752288Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T21:56:39.015 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.948460588Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.949176464Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.9495632Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.950291026Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.950639346Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.951044444Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.951459117Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.951860252Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.952237224Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.952603754Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.953362907Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.953733691Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.954130458Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.954511092Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.954894585Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.955149963Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T21:56:39.016 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.955380153Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.955769029Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.956187852Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.956585729Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.956829366Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.957239845Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.957594271Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.958026032Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.958455705Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.959242127Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.959655015Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.959950474Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.960355798Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.960731068Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T21:56:39.017 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.961125455Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.961414391Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.961772576Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.962219531Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.962570285Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.962987217Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.963752901Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.96415909Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.964523453Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.967909138Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.971231012Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.971661956Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.972109031Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T21:56:39.018 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.972891729Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.973291621Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.973695344Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.974110677Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.974514272Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.97473577Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.975502349Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.976336142Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.977152226Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.977545044Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.977954027Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.978746463Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.979556113Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.979962122Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.980745052Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.981553443Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.981809752Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T21:56:39.019 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.982267062Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.982667731Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.983076863Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.983332679Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.984164126Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.984987348Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.985762674Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.986569237Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.9874206Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.987680995Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.988042108Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.988824755Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.989085839Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.989942186Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.990341747Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.991140917Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.991558848Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T21:56:39.020 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.992030893Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.992973414Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.993378015Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.993806914Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.994189826Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.994601851Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.994901475Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.995306143Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.995710061Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.995814293Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.996095807Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.996434903Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.996971807Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.99714309Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.997300148Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.997804521Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.998370808Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T21:56:39.021 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.998939143Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:38.999452238Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.000042782Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.000626889Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.000988878Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.001457999Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.00200612Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.002557617Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.010517349Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.011908997Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T21:56:39.022 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.012286891Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T21:56:39.489 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.017129484Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T21:56:39.489 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.021297923Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T21:56:39.489 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.021669047Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T21:56:39.489 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.022076406Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T21:56:39.489 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.022376207Z level=info msg="Executing migration" id="create permission table" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.02272527Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.023129599Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.02353804Z level=info msg="Executing migration" id="create role table" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.023904243Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.024774352Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.025676617Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.026132417Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.026529644Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.026927605Z level=info msg="Executing migration" id="create team role table" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.027282777Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.027693624Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.028128635Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.028520418Z level=info msg="Executing migration" id="create user role table" 2024-03-11T21:56:39.490 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.028910493Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T21:56:39.491 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.029357608Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T21:56:39.491 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.029779924Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T21:56:39.491 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.030224942Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T21:56:39.491 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.030614357Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T21:56:39.491 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.031047614Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.031486881Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.032442848Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.032903101Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.033331903Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.033702082Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.034115465Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.034474987Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.034898486Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.035824023Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.036203725Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.036620209Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T21:56:39.492 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.036906996Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.037337019Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.037749061Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.038115729Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.038372855Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.038664007Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.03903256Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.039447356Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.040431768Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.040713466Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.04114623Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.041557174Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.041987955Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.042938709Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.043335773Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T21:56:39.493 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.043716319Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.043812924Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.044102282Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.044211449Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.044488977Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.044848062Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.04528477Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.045711659Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.046213482Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.046596635Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.046984304Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.047340573Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.047693052Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:56:39.494 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.048108782Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.048530751Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.048931214Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.051922549Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.052892182Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.053844435Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.054130223Z level=info msg="Executing migration" id="add share column" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.055095767Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.055478909Z level=info msg="Executing migration" id="create file table" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.05606716Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.056465872Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T21:56:39.495 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.056902187Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T21:56:39.496 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.057251033Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T21:56:39.496 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.057637206Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T21:56:39.496 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.057892766Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T21:56:39.496 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.058215254Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T21:56:39.496 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.058455274Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T21:56:39.496 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.058805055Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T21:56:39.496 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.05976111Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.060048972Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.060464863Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.06080314Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.061123571Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.061426168Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.062442239Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.063456473Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.063869538Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.073491535Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.073931817Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.074402132Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.077725617Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.078040621Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.078358917Z level=info msg="Executing migration" id="create folder table" 2024-03-11T21:56:39.497 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.07869889Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.079146974Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.079561396Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.07980925Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=migrator t=2024-03-11T21:56:39.080219668Z level=info msg="migrations completed" performed=485 skipped=0 duration=249.67589ms 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=sqlstore t=2024-03-11T21:56:39.080614499Z level=info msg="Created default organization" 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=plugin.loader t=2024-03-11T21:56:39.136167246Z level=info msg="Plugin registered" pluginID=input 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=plugin.loader t=2024-03-11T21:56:39.163012674Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=plugin.loader t=2024-03-11T21:56:39.163031736Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=secrets t=2024-03-11T21:56:39.16308701Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=query_data t=2024-03-11T21:56:39.165165596Z level=info msg="Query Service initialization" 2024-03-11T21:56:39.498 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=live.push_http t=2024-03-11T21:56:39.167570808Z level=info msg="Live Push Gateway initialization" 2024-03-11T21:56:40.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[25338]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:40.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[34177]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:39 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.171 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:39 smithi102 systemd[1]: Starting Ceph node-exporter.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:56:40.228 INFO:teuthology.orchestra.run.smithi102.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T21:56:40.228 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-11T21:56:40.228 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme1n1 ssd Linux_fe33302851940c7a99cb 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:56:40.228 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme2n1 ssd Linux_e418c6793f1896d0b0d3 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:56:40.228 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme3n1 ssd Linux_c4bc35a5e15294c2d117 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:56:40.228 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme4n1 ssd Linux_2008554550ed85afad45 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:56:40.228 INFO:teuthology.orchestra.run.smithi102.stdout:smithi110 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002T480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-11T21:56:40.228 INFO:teuthology.orchestra.run.smithi102.stdout:smithi110 /dev/nvme1n1 ssd Linux_57ede36057af39640ba0 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:56:40.228 INFO:teuthology.orchestra.run.smithi102.stdout:smithi110 /dev/nvme2n1 ssd Linux_8dd66951b051d85857b8 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:56:40.229 INFO:teuthology.orchestra.run.smithi102.stdout:smithi110 /dev/nvme3n1 ssd Linux_0f094762ce4cede9c919 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:56:40.229 INFO:teuthology.orchestra.run.smithi102.stdout:smithi110 /dev/nvme4n1 ssd Linux_8160aeb7118ce9398ce1 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:56:40.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-mon[30535]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:40.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:40.239 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=infra.usagestats.collector t=2024-03-11T21:56:39.770530885Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T21:56:40.239 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=provisioning.datasources t=2024-03-11T21:56:39.771521647Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T21:56:40.239 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=provisioning.datasources t=2024-03-11T21:56:39.782272248Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T21:56:40.239 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=provisioning.alerting t=2024-03-11T21:56:39.793033546Z level=info msg="starting to provision alerting" 2024-03-11T21:56:40.239 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=provisioning.alerting t=2024-03-11T21:56:39.793084564Z level=info msg="finished to provision alerting" 2024-03-11T21:56:40.239 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=ngalert.state.manager t=2024-03-11T21:56:39.793226639Z level=info msg="Warming state cache for startup" 2024-03-11T21:56:40.240 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=grafanaStorageLogger t=2024-03-11T21:56:39.793224151Z level=info msg="storage starting" 2024-03-11T21:56:40.240 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=http.server t=2024-03-11T21:56:39.795629825Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T21:56:40.240 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=ngalert.state.manager t=2024-03-11T21:56:39.804974699Z level=info msg="State cache has been initialized" states=0 duration=11.746987ms 2024-03-11T21:56:40.240 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=ticker t=2024-03-11T21:56:39.805050563Z level=info msg=starting first_tick=2024-03-11T21:56:40Z 2024-03-11T21:56:40.240 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=ngalert.multiorg.alertmanager t=2024-03-11T21:56:39.805049819Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T21:56:40.240 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:56:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a[70105]: logger=sqlstore.transactions t=2024-03-11T21:56:39.845123558Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:56:40.550 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:40 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ::ffff:172.21.15.110 - - [11/Mar/2024:21:56:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:56:40.550 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:40 smithi102 bash[79776]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T21:56:40.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-11T21:56:41.111 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:40 smithi102 ceph-mon[34177]: Deploying daemon node-exporter.a on smithi102 2024-03-11T21:56:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:40 smithi102 ceph-mon[25338]: Deploying daemon node-exporter.a on smithi102 2024-03-11T21:56:41.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:40 smithi110 ceph-mon[30535]: Deploying daemon node-exporter.a on smithi102 2024-03-11T21:56:41.813 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:41 smithi102 bash[79776]: Getting image source signatures 2024-03-11T21:56:41.813 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:41 smithi102 bash[79776]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T21:56:41.814 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:41 smithi102 bash[79776]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T21:56:41.814 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:41 smithi102 bash[79776]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T21:56:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:41 smithi102 ceph-mon[25338]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:41 smithi102 ceph-mon[25338]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:41 smithi102 ceph-mon[34177]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:41 smithi102 ceph-mon[34177]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:42.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:41 smithi110 ceph-mon[30535]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:42.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:41 smithi110 ceph-mon[30535]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:43.160 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: alertmanager 2024-03-11T21:56:43.160 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: alertmanager 2024-03-11T21:56:43.160 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-03-11T21:56:43.160 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-03-11T21:56:43.160 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102=a 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-03-11T21:55:08.658010Z' 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout: - 9093 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout: - 9094 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout: running: 0 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout:events: 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout:- 2024-03-11T21:56:17.662395Z service:alertmanager [INFO] "service was created" 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: grafana 2024-03-11T21:56:43.161 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: grafana 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi110=a 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout:spec: 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout: anonymous_access: true 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout: protocol: https 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-03-11T21:55:12.441578Z' 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-03-11T21:56:43.162 INFO:teuthology.orchestra.run.smithi102.stdout: - 3000 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout: running: 0 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout:events: 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout:- 2024-03-11T21:56:38.870602Z service:grafana [INFO] "service was created" 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: iscsi 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout:service_id: datapool 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: iscsi.datapool 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-03-11T21:56:43.163 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi110=iscsi.a 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout:spec: 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: api_password: admin 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: api_port: 5000 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: api_user: admin 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: pool: datapool 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: trusted_ip_list: 172.21.15.110 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-03-11T21:54:57.168847Z' 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-03-11T21:55:35.040167Z' 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: - 5000 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: running: 1 2024-03-11T21:56:43.164 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: mgr 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: mgr 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout: count: 2 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102=y 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi110=x 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-03-11T21:51:05.452231Z' 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-03-11T21:55:35.039783Z' 2024-03-11T21:56:43.165 INFO:teuthology.orchestra.run.smithi102.stdout: running: 2 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout: size: 2 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: mon 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: mon 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout: count: 3 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102:172.21.15.102=a 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi110:172.21.15.110=b 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-03-11T21:50:41.259811Z' 2024-03-11T21:56:43.166 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-03-11T21:55:35.039679Z' 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout: running: 3 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout: size: 3 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: node-exporter 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: node-exporter 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout: count: 2 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102=a 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi110=b 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-03-11T21:55:04.516923Z' 2024-03-11T21:56:43.167 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout: - 9100 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout: running: 0 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout: size: 2 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: osd 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: osd 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout:unmanaged: true 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout:spec: 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout: filter_logic: AND 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout: objectstore: bluestore 2024-03-11T21:56:43.168 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout: container_image_id: a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-03-11T21:55:35.039859Z' 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout: running: 8 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout: size: 8 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: prometheus 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: prometheus 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi110=a 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-03-11T21:56:43.169 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-03-11T21:55:00.608160Z' 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-03-11T21:55:35.040240Z' 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout: - 9095 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout: running: 1 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: rgw 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout:service_id: foo.a 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: rgw.foo.a 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-03-11T21:56:43.170 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-03-11T21:56:43.171 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102=foo.a 2024-03-11T21:56:43.171 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-03-11T21:56:43.171 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-03-11T21:54:49.140745Z' 2024-03-11T21:56:43.171 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-03-11T21:55:58.987476Z' 2024-03-11T21:56:43.171 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-03-11T21:56:43.171 INFO:teuthology.orchestra.run.smithi102.stdout: - 80 2024-03-11T21:56:43.171 INFO:teuthology.orchestra.run.smithi102.stdout: running: 1 2024-03-11T21:56:43.171 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-03-11T21:56:43.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:42 smithi102 ceph-mon[25338]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:56:43.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:42 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:43.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:42 smithi102 ceph-mon[34177]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:56:43.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:42 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:43.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:42 smithi110 ceph-mon[30535]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:56:43.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:42 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:43.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:56:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:56:43.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-11T21:56:44.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:43 smithi110 ceph-mon[30535]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:44.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:43 smithi110 ceph-mon[30535]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T21:56:44.354 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:44.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:43 smithi102 ceph-mon[25338]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:44.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:43 smithi102 ceph-mon[25338]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T21:56:44.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:43 smithi102 ceph-mon[34177]: from='client.14523 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:56:44.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:43 smithi102 ceph-mon[34177]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T21:56:44.669 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:44 smithi102 bash[79776]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T21:56:44.669 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:44 smithi102 bash[79776]: Writing manifest to image destination 2024-03-11T21:56:44.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:44 smithi102 ceph-mon[25338]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:44.997 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:44 smithi102 ceph-mon[34177]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:45.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:44 smithi110 ceph-mon[30535]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:56:45.358 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 podman[79776]: 2024-03-11 21:56:45.132940131 +0000 UTC m=+5.039671773 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T21:56:45.358 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 podman[79776]: 2024-03-11 21:56:45.365387759 +0000 UTC m=+5.272119437 container create 39b53abb1c4d13b853c10accafb7c9971f305d348986542110aaf7e26da4e2cc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:56:45.609 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 podman[79776]: 2024-03-11 21:56:45.609455206 +0000 UTC m=+5.516186847 container init 39b53abb1c4d13b853c10accafb7c9971f305d348986542110aaf7e26da4e2cc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:56:45.609 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 podman[79776]: 2024-03-11 21:56:45.612180524 +0000 UTC m=+5.518912168 container start 39b53abb1c4d13b853c10accafb7c9971f305d348986542110aaf7e26da4e2cc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:56:45.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.616Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T21:56:45.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.616Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T21:56:45.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.616Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T21:56:45.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.616Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T21:56:45.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.617Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T21:56:45.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T21:56:45.925 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T21:56:45.926 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T21:56:45.926 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T21:56:45.926 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T21:56:45.926 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T21:56:45.926 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T21:56:45.926 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a[80091]: ts=2024-03-11T21:56:45.618Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T21:56:45.926 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 bash[79776]: 39b53abb1c4d13b853c10accafb7c9971f305d348986542110aaf7e26da4e2cc 2024-03-11T21:56:45.926 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:56:45 smithi102 systemd[1]: Started Ceph node-exporter.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:56:46.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:46 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:46.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:46 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:46.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:46 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:46.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:46 smithi102 ceph-mon[25338]: Deploying daemon node-exporter.b on smithi110 2024-03-11T21:56:46.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:46 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:46.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:46 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:46.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:46 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:46.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:46 smithi102 ceph-mon[34177]: Deploying daemon node-exporter.b on smithi110 2024-03-11T21:56:47.105 INFO:teuthology.orchestra.run.smithi102.stdout:osd 8 72s ago - 2024-03-11T21:56:47.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:46 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:47.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:46 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:47.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:46 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' 2024-03-11T21:56:47.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:46 smithi110 ceph-mon[30535]: Deploying daemon node-exporter.b on smithi110 2024-03-11T21:56:47.239 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:46 smithi110 bash[70359]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T21:56:47.738 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:47 smithi110 bash[70359]: Getting image source signatures 2024-03-11T21:56:47.739 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:47 smithi110 bash[70359]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T21:56:47.739 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:47 smithi110 bash[70359]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T21:56:47.739 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:47 smithi110 bash[70359]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T21:56:47.823 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T21:56:47.834 INFO:tasks.cephadm:Teardown begin 2024-03-11T21:56:47.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:47.866 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:47.899 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T21:56:47.899 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:56:47.920 DEBUG:teuthology.orchestra.run.smithi110:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:56:47.954 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T21:56:47.955 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T21:56:47.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:56:48.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:47 smithi102 ceph-mon[34177]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:48.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:47 smithi102 ceph-mon[34177]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:47 smithi102 ceph-mon[25338]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:47 smithi102 ceph-mon[25338]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:48 smithi102 systemd[1]: Stopping Ceph mon.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:56:48.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:47 smithi110 ceph-mon[30535]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:56:48.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:47 smithi110 ceph-mon[30535]: from='mgr.14559 172.21.15.102:0/2423851471' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:56:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:48 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a[25334]: 2024-03-11T21:56:48.291+0000 7fc44d9ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:56:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:48 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a[25334]: 2024-03-11T21:56:48.291+0000 7fc44d9ce700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T21:56:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:48 smithi102 podman[80265]: 2024-03-11 21:56:48.383987143 +0000 UTC m=+0.226125677 container died e26e085d973f9870f4d1f248cfadf9e5bb30557be480efd2e462801928471879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T21:56:48.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:48 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:48] ENGINE Bus STOPPING 2024-03-11T21:56:48.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:48 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:56:48.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:48 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:48] ENGINE Bus STOPPED 2024-03-11T21:56:48.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:48 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:48] ENGINE Bus STARTING 2024-03-11T21:56:48.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:48 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:48] ENGINE Serving on http://:::9283 2024-03-11T21:56:48.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:48 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:48] ENGINE Bus STARTED 2024-03-11T21:56:49.095 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:48 smithi102 podman[80265]: 2024-03-11 21:56:48.867413915 +0000 UTC m=+0.709552490 container cleanup e26e085d973f9870f4d1f248cfadf9e5bb30557be480efd2e462801928471879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T21:56:49.095 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 11 21:56:48 smithi102 bash[80265]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:56:49.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T21:56:49.267 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:56:49.268 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T21:56:49.268 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-11T21:56:49.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.c 2024-03-11T21:56:49.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:49 smithi102 systemd[1]: Stopping Ceph mon.c for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:56:49.890 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:49 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-c[34173]: 2024-03-11T21:56:49.562+0000 7f2e387ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:56:49.890 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:49 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-c[34173]: 2024-03-11T21:56:49.562+0000 7f2e387ef700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:56:49.891 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:49 smithi102 podman[80396]: 2024-03-11 21:56:49.614429035 +0000 UTC m=+0.164275191 container died d65f8f2d5f2ef62ff315196447cc826cb3d6b6bb1927c9e0674e96fd9f6af839 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T21:56:49.891 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:49 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:49] ENGINE Bus STOPPING 2024-03-11T21:56:49.891 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:49 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:56:49.891 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:49 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:49] ENGINE Bus STOPPED 2024-03-11T21:56:49.891 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:49 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:49] ENGINE Bus STARTING 2024-03-11T21:56:49.891 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:49 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:49] ENGINE Serving on http://:::9283 2024-03-11T21:56:49.891 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:49 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: [11/Mar/2024:21:56:49] ENGINE Bus STARTED 2024-03-11T21:56:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:50 smithi102 podman[80396]: 2024-03-11 21:56:50.071911973 +0000 UTC m=+0.621758170 container cleanup d65f8f2d5f2ef62ff315196447cc826cb3d6b6bb1927c9e0674e96fd9f6af839 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T21:56:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:50 smithi102 bash[80396]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-c 2024-03-11T21:56:50.465 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:50 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: ::ffff:172.21.15.110 - - [11/Mar/2024:21:56:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:56:50.466 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 11 21:56:50 smithi102 podman[80421]: 2024-03-11 21:56:50.356779746 +0000 UTC m=+0.743846215 container remove d65f8f2d5f2ef62ff315196447cc826cb3d6b6bb1927c9e0674e96fd9f6af839 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T21:56:50.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T21:56:50.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:56:50.516 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-11T21:56:50.517 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-11T21:56:50.517 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.b 2024-03-11T21:56:50.838 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:50 smithi110 systemd[1]: Stopping Ceph mon.b for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:56:51.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:51 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-b[30531]: 2024-03-11T21:56:51.705+0000 7f05e5de5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:56:51.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:51 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-b[30531]: 2024-03-11T21:56:51.705+0000 7f05e5de5700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:56:51.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:51 smithi110 podman[70453]: 2024-03-11 21:56:51.736815418 +0000 UTC m=+1.024760874 container died 6ebd0e2e1fe255fe4e5bd4273c51fe9f6b32b57b8a7ac7ba4ee561d0a879237a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-11T21:56:53.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:56:52 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug there is no tcmu-runner data available 2024-03-11T21:56:55.899 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:55 smithi110 bash[70359]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T21:56:55.899 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:55 smithi110 podman[70453]: 2024-03-11 21:56:55.662858287 +0000 UTC m=+4.950803743 container cleanup 6ebd0e2e1fe255fe4e5bd4273c51fe9f6b32b57b8a7ac7ba4ee561d0a879237a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-11T21:56:55.899 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:55 smithi110 bash[70453]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-b 2024-03-11T21:56:56.211 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:55 smithi110 podman[70468]: 2024-03-11 21:56:55.948364746 +0000 UTC m=+4.208595751 container remove 6ebd0e2e1fe255fe4e5bd4273c51fe9f6b32b57b8a7ac7ba4ee561d0a879237a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mon-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-03-11T21:56:56.211 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:55 smithi110 bash[70359]: Writing manifest to image destination 2024-03-11T21:56:56.211 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 podman[70359]: 2024-03-11 21:56:56.127877184 +0000 UTC m=+9.230769589 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T21:56:56.211 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 podman[70359]: 2024-03-11 21:56:56.218129592 +0000 UTC m=+9.321021980 container create 56a6f1570c42c3c7c6e1d41aa9569863e035b0817e06c244c6c3c31b02ca569a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:56:56.440 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T21:56:56.464 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:56 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-11T21:56:56.464 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:56 smithi110 systemd[1]: Stopped Ceph mon.b for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:56:56.464 INFO:journalctl@ceph.mon.b.smithi110.stdout:Mar 11 21:56:56 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mon.b.service: Consumed 5.245s CPU time. 2024-03-11T21:56:56.465 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 podman[70359]: 2024-03-11 21:56:56.428848529 +0000 UTC m=+9.531740933 container init 56a6f1570c42c3c7c6e1d41aa9569863e035b0817e06c244c6c3c31b02ca569a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:56:56.465 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 podman[70359]: 2024-03-11 21:56:56.431561822 +0000 UTC m=+9.534454220 container start 56a6f1570c42c3c7c6e1d41aa9569863e035b0817e06c244c6c3c31b02ca569a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:56:56.465 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.435Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T21:56:56.465 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.435Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T21:56:56.465 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.435Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T21:56:56.465 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.435Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T21:56:56.465 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.435Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.435Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T21:56:56.466 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T21:56:56.467 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T21:56:56.468 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T21:56:56.469 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T21:56:56.469 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T21:56:56.469 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T21:56:56.469 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T21:56:56.469 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T21:56:56.469 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b[70542]: ts=2024-03-11T21:56:56.436Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T21:56:56.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:56:56.512 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-11T21:56:56.512 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T21:56:56.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:56:56.739 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 bash[70359]: 56a6f1570c42c3c7c6e1d41aa9569863e035b0817e06c244c6c3c31b02ca569a 2024-03-11T21:56:56.739 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:56:56 smithi110 systemd[1]: Started Ceph node-exporter.b for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:56:56.796 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:56 smithi102 systemd[1]: Stopping Ceph mgr.y for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:56:57.114 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:56 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:56:56.801+0000 7f7ee3911700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:56:57.114 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:56 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y[25564]: 2024-03-11T21:56:56.801+0000 7f7ee3911700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T21:56:57.114 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:56 smithi102 podman[80527]: 2024-03-11 21:56:56.834362158 +0000 UTC m=+0.129962421 container died e2bcbbac623f8f530c812e0d0c41a92e6192452f012c94a62122a42e4c0a9e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:56:57.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:57 smithi102 podman[80527]: 2024-03-11 21:56:57.295231291 +0000 UTC m=+0.590831555 container cleanup e2bcbbac623f8f530c812e0d0c41a92e6192452f012c94a62122a42e4c0a9e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T21:56:57.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 11 21:56:57 smithi102 bash[80527]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-y 2024-03-11T21:56:57.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T21:56:57.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:56:57.792 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T21:56:57.792 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T21:56:57.792 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.x 2024-03-11T21:56:58.113 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:56:57 smithi110 systemd[1]: Stopping Ceph mgr.x for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:56:58.384 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:56:58 smithi110 podman[70625]: 2024-03-11 21:56:58.12013235 +0000 UTC m=+0.121698621 container died 41c8f46117c1615567965992391e462f4fb18d6dd0dbf6e56f1d4730a7f56b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x, RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-11T21:56:58.738 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:56:58 smithi110 podman[70625]: 2024-03-11 21:56:58.491213304 +0000 UTC m=+0.492779582 container cleanup 41c8f46117c1615567965992391e462f4fb18d6dd0dbf6e56f1d4730a7f56b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T21:56:58.739 INFO:journalctl@ceph.mgr.x.smithi110.stdout:Mar 11 21:56:58 smithi110 bash[70625]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-mgr-x 2024-03-11T21:56:58.945 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T21:56:58.982 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:56:58.982 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T21:56:58.983 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T21:56:58.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.0 2024-03-11T21:56:59.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:56:59 smithi102 systemd[1]: Stopping Ceph osd.0 for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:56:59.919 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:56:59 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0[38048]: 2024-03-11T21:56:59.467+0000 7f05f99e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:56:59.919 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:56:59 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0[38048]: 2024-03-11T21:56:59.467+0000 7f05f99e0700 -1 osd.0 72 *** Got signal Terminated *** 2024-03-11T21:56:59.919 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:56:59 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0[38048]: 2024-03-11T21:56:59.467+0000 7f05f99e0700 -1 osd.0 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:57:05.053 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:04 smithi102 podman[80648]: 2024-03-11 21:57:04.780599735 +0000 UTC m=+5.603069374 container died 7d7ec9520d008be8f5562509fb45f4a487d9aac626d536d950905cc41ae2a7a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T21:57:05.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:05 smithi102 podman[80648]: 2024-03-11 21:57:05.275463333 +0000 UTC m=+6.097932971 container cleanup 7d7ec9520d008be8f5562509fb45f4a487d9aac626d536d950905cc41ae2a7a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-11T21:57:05.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:05 smithi102 bash[80648]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0 2024-03-11T21:57:05.817 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:05 smithi102 podman[80665]: 2024-03-11 21:57:05.523212542 +0000 UTC m=+0.738426570 container remove 7d7ec9520d008be8f5562509fb45f4a487d9aac626d536d950905cc41ae2a7a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T21:57:06.119 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:05 smithi102 podman[80756]: 2024-03-11 21:57:05.823931789 +0000 UTC m=+0.103983323 container create ea2d905a7f876cbc06597c325584ed40a898db17f8d585cfb04c638475a33a0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T21:57:06.119 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:05 smithi102 podman[80756]: 2024-03-11 21:57:05.737874137 +0000 UTC m=+0.017925671 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:06.119 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:06 smithi102 podman[80756]: 2024-03-11 21:57:06.034724435 +0000 UTC m=+0.314775980 container init ea2d905a7f876cbc06597c325584ed40a898db17f8d585cfb04c638475a33a0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-11T21:57:06.119 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:06 smithi102 podman[80756]: 2024-03-11 21:57:06.038207082 +0000 UTC m=+0.318258622 container start ea2d905a7f876cbc06597c325584ed40a898db17f8d585cfb04c638475a33a0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-11T21:57:06.119 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:06 smithi102 podman[80756]: 2024-03-11 21:57:06.126245147 +0000 UTC m=+0.406296702 container attach ea2d905a7f876cbc06597c325584ed40a898db17f8d585cfb04c638475a33a0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:57:06.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 11 21:57:06 smithi102 podman[80756]: 2024-03-11 21:57:06.325936693 +0000 UTC m=+0.605988228 container died ea2d905a7f876cbc06597c325584ed40a898db17f8d585cfb04c638475a33a0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T21:57:06.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T21:57:07.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:57:07.036 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T21:57:07.036 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T21:57:07.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.1 2024-03-11T21:57:07.360 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:07 smithi102 systemd[1]: Stopping Ceph osd.1 for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:57:07.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1[45431]: 2024-03-11T21:57:07.366+0000 7facf6771700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:57:07.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1[45431]: 2024-03-11T21:57:07.366+0000 7facf6771700 -1 osd.1 72 *** Got signal Terminated *** 2024-03-11T21:57:07.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:07 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1[45431]: 2024-03-11T21:57:07.366+0000 7facf6771700 -1 osd.1 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:57:13.093 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:12 smithi102 podman[80863]: 2024-03-11 21:57:12.829539953 +0000 UTC m=+5.560343979 container died 1c55d0dd19a06d24d9d4ccb60d7fc11de7a2192386523a780935e7919a534693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T21:57:13.419 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:13 smithi102 podman[80863]: 2024-03-11 21:57:13.266050395 +0000 UTC m=+5.996854430 container cleanup 1c55d0dd19a06d24d9d4ccb60d7fc11de7a2192386523a780935e7919a534693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD) 2024-03-11T21:57:13.419 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:13 smithi102 bash[80863]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1 2024-03-11T21:57:13.861 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:13 smithi102 podman[80880]: 2024-03-11 21:57:13.459078918 +0000 UTC m=+0.623401047 container remove 1c55d0dd19a06d24d9d4ccb60d7fc11de7a2192386523a780935e7919a534693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T21:57:14.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:13 smithi102 podman[80973]: 2024-03-11 21:57:13.868104548 +0000 UTC m=+0.109869706 container create ae1f4e8950c0ca9f13e9d3dfa984ad524463720d9f07351baaa3813b9994f0cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:57:14.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:13 smithi102 podman[80973]: 2024-03-11 21:57:13.776081336 +0000 UTC m=+0.017846504 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:14.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:14 smithi102 podman[80973]: 2024-03-11 21:57:14.096124971 +0000 UTC m=+0.337890129 container init ae1f4e8950c0ca9f13e9d3dfa984ad524463720d9f07351baaa3813b9994f0cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T21:57:14.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:14 smithi102 podman[80973]: 2024-03-11 21:57:14.099191485 +0000 UTC m=+0.340956649 container start ae1f4e8950c0ca9f13e9d3dfa984ad524463720d9f07351baaa3813b9994f0cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T21:57:14.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:14 smithi102 podman[80973]: 2024-03-11 21:57:14.178848887 +0000 UTC m=+0.420614080 container attach ae1f4e8950c0ca9f13e9d3dfa984ad524463720d9f07351baaa3813b9994f0cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T21:57:14.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 11 21:57:14 smithi102 podman[80973]: 2024-03-11 21:57:14.391198209 +0000 UTC m=+0.632963369 container died ae1f4e8950c0ca9f13e9d3dfa984ad524463720d9f07351baaa3813b9994f0cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T21:57:15.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T21:57:15.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:57:15.055 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T21:57:15.056 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T21:57:15.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.2 2024-03-11T21:57:15.379 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:15 smithi102 systemd[1]: Stopping Ceph osd.2 for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:57:15.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:15 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2[53625]: 2024-03-11T21:57:15.384+0000 7fb0ed5e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:57:15.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:15 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2[53625]: 2024-03-11T21:57:15.384+0000 7fb0ed5e5700 -1 osd.2 72 *** Got signal Terminated *** 2024-03-11T21:57:15.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:15 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2[53625]: 2024-03-11T21:57:15.384+0000 7fb0ed5e5700 -1 osd.2 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:57:21.021 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:20 smithi102 podman[81079]: 2024-03-11 21:57:20.744109158 +0000 UTC m=+5.464595442 container stop bc79c903fefbfb4aac407747f8123998c7691cfb3e3cb9a0645cf506d7401546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T21:57:21.021 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:20 smithi102 podman[81079]: 2024-03-11 21:57:20.773518178 +0000 UTC m=+5.494004516 container died bc79c903fefbfb4aac407747f8123998c7691cfb3e3cb9a0645cf506d7401546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T21:57:21.380 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:21 smithi102 podman[81079]: 2024-03-11 21:57:21.219100548 +0000 UTC m=+5.939586824 container cleanup bc79c903fefbfb4aac407747f8123998c7691cfb3e3cb9a0645cf506d7401546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T21:57:21.380 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:21 smithi102 bash[81079]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2 2024-03-11T21:57:21.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:21 smithi102 podman[81108]: 2024-03-11 21:57:21.38709973 +0000 UTC m=+0.635530954 container remove bc79c903fefbfb4aac407747f8123998c7691cfb3e3cb9a0645cf506d7401546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T21:57:22.125 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:21 smithi102 podman[81195]: 2024-03-11 21:57:21.728692383 +0000 UTC m=+0.017508228 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:22.125 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:21 smithi102 podman[81195]: 2024-03-11 21:57:21.829454957 +0000 UTC m=+0.118270794 container create a76d0b508ef0a78cfb5383135cafb0dcc1d43fae53214002a6941ab680518e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T21:57:22.125 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:22 smithi102 podman[81195]: 2024-03-11 21:57:22.048637917 +0000 UTC m=+0.337453757 container init a76d0b508ef0a78cfb5383135cafb0dcc1d43fae53214002a6941ab680518e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397) 2024-03-11T21:57:22.125 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:22 smithi102 podman[81195]: 2024-03-11 21:57:22.051761087 +0000 UTC m=+0.340576926 container start a76d0b508ef0a78cfb5383135cafb0dcc1d43fae53214002a6941ab680518e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-11T21:57:22.125 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:22 smithi102 podman[81195]: 2024-03-11 21:57:22.13191621 +0000 UTC m=+0.420732065 container attach a76d0b508ef0a78cfb5383135cafb0dcc1d43fae53214002a6941ab680518e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T21:57:22.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 11 21:57:22 smithi102 podman[81195]: 2024-03-11 21:57:22.349653591 +0000 UTC m=+0.638469440 container died a76d0b508ef0a78cfb5383135cafb0dcc1d43fae53214002a6941ab680518e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397) 2024-03-11T21:57:23.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T21:57:23.095 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:57:23.095 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T21:57:23.096 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T21:57:23.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.3 2024-03-11T21:57:23.419 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:23 smithi102 systemd[1]: Stopping Ceph osd.3 for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:57:23.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:23 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3[62365]: 2024-03-11T21:57:23.429+0000 7f362a7ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:57:23.920 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:23 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3[62365]: 2024-03-11T21:57:23.429+0000 7f362a7ee700 -1 osd.3 72 *** Got signal Terminated *** 2024-03-11T21:57:23.920 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:23 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3[62365]: 2024-03-11T21:57:23.429+0000 7f362a7ee700 -1 osd.3 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:57:23.989 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:57:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:57:23.661Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-03-11T21:57:23.989 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:57:23 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:57:23.663Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-03-11T21:57:25.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:25 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3[62365]: 2024-03-11T21:57:25.519+0000 7f3622dda700 -1 osd.3 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:56:59.219621+0000 front 2024-03-11T21:56:59.219529+0000 (oldest deadline 2024-03-11T21:57:25.119699+0000) 2024-03-11T21:57:26.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:26 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3[62365]: 2024-03-11T21:57:26.539+0000 7f3622dda700 -1 osd.3 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:56:59.219621+0000 front 2024-03-11T21:56:59.219529+0000 (oldest deadline 2024-03-11T21:57:25.119699+0000) 2024-03-11T21:57:26.957 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:26.627+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:27.239 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:27 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:27.072+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:27.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:26 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:26.963+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:27.739 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:27 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:27.601+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:27.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:27 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:27.405+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:27.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:27 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3[62365]: 2024-03-11T21:57:27.503+0000 7f3622dda700 -1 osd.3 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:56:59.219621+0000 front 2024-03-11T21:56:59.219529+0000 (oldest deadline 2024-03-11T21:57:25.119699+0000) 2024-03-11T21:57:28.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:27 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:27.972+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:28.239 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:28 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:28.073+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:28.739 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:28 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:28.627+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:28.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:28 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:28.414+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:29.141 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:28 smithi102 podman[81298]: 2024-03-11 21:57:28.892542676 +0000 UTC m=+5.569471404 container died fee6d2682609a4fc31599974e23ac082d901f0fc53328148ad68ca0f17ef9d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T21:57:29.239 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:29 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:29.114+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:29.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:28 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:28.947+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:29.419 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:29 smithi102 podman[81298]: 2024-03-11 21:57:29.313580924 +0000 UTC m=+5.990509697 container cleanup fee6d2682609a4fc31599974e23ac082d901f0fc53328148ad68ca0f17ef9d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T21:57:29.419 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:29 smithi102 bash[81298]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3 2024-03-11T21:57:29.738 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:29 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:29.422+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:29.739 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:29 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:29.643+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:29.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:29 smithi102 podman[81316]: 2024-03-11 21:57:29.523242197 +0000 UTC m=+0.623928888 container remove fee6d2682609a4fc31599974e23ac082d901f0fc53328148ad68ca0f17ef9d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T21:57:30.181 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:29 smithi102 podman[81407]: 2024-03-11 21:57:29.835608557 +0000 UTC m=+0.018435619 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:30.182 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:29 smithi102 podman[81407]: 2024-03-11 21:57:29.964650051 +0000 UTC m=+0.147477142 container create 16eeec1dd9fb2cf69296344479a320fbed57d107d89c4b8784e653b597c51f5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-11T21:57:30.238 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:30 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:30.091+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:30.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:29 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:29.920+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:30.587 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:30 smithi102 podman[81407]: 2024-03-11 21:57:30.293127764 +0000 UTC m=+0.475954818 container init 16eeec1dd9fb2cf69296344479a320fbed57d107d89c4b8784e653b597c51f5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:57:30.587 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:30 smithi102 podman[81407]: 2024-03-11 21:57:30.296149108 +0000 UTC m=+0.478976159 container start 16eeec1dd9fb2cf69296344479a320fbed57d107d89c4b8784e653b597c51f5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-11T21:57:30.587 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:30 smithi102 podman[81407]: 2024-03-11 21:57:30.376261221 +0000 UTC m=+0.559088290 container attach 16eeec1dd9fb2cf69296344479a320fbed57d107d89c4b8784e653b597c51f5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T21:57:30.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:30 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:30.426+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:30.739 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:30 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:30.602+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:30.881 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 11 21:57:30 smithi102 podman[81407]: 2024-03-11 21:57:30.593776685 +0000 UTC m=+0.776603736 container died 16eeec1dd9fb2cf69296344479a320fbed57d107d89c4b8784e653b597c51f5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-3-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-11T21:57:31.239 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:31.088+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:31.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:30 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:30.930+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:31.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T21:57:31.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:57:31.282 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T21:57:31.282 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T21:57:31.283 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.4 2024-03-11T21:57:31.598 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:31 smithi110 systemd[1]: Stopping Ceph osd.4 for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:57:31.598 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:31.458+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:31.894 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:31.603+0000 7f379a32b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:57:31.895 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:31.603+0000 7f379a32b700 -1 osd.4 72 *** Got signal Terminated *** 2024-03-11T21:57:31.895 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:31.603+0000 7f379a32b700 -1 osd.4 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:57:31.895 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:31.633+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:32.238 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:32 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:32.126+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:32.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:31 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:31.900+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:32.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:32 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:32.465+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:32.739 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:32 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:32.675+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:33.239 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:33 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:33.161+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:33.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:32 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:32.916+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:33.704 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:33 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:33.418+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:33.989 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:33 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:33.710+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:33.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:33 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:33.954+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:34.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:34 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:34.205+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:34.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:34 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:34.383+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:34.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:34 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:34.911+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:34.989 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:34 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:34.680+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:34.989 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:34 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:34.680+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.552003+0000 front 2024-03-11T21:57:11.551936+0000 (oldest deadline 2024-03-11T21:57:34.451933+0000) 2024-03-11T21:57:35.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:35 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:35.373+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:35.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:35 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:35.373+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:35.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:35 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:35.190+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:35.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:35 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:35.190+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:35.989 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:35 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:35.657+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:02.151186+0000 front 2024-03-11T21:57:02.151356+0000 (oldest deadline 2024-03-11T21:57:26.251105+0000) 2024-03-11T21:57:35.989 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:35 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4[35434]: 2024-03-11T21:57:35.657+0000 7f379691f700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.552003+0000 front 2024-03-11T21:57:11.551936+0000 (oldest deadline 2024-03-11T21:57:34.451933+0000) 2024-03-11T21:57:35.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:35 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:35.861+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:35.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:35 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:35.861+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:36.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:36 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:36.218+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:36.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:36 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:36.218+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:36.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:36 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:36.323+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:36.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:36 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:36.323+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:37.121 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:36 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:36.860+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:37.121 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:36 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:36.860+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:37.390 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:37 smithi110 podman[70765]: 2024-03-11 21:57:37.127817781 +0000 UTC m=+5.625821773 container died b6a67b672a5b39a581aa43e9cc5f45b616cb5819f3599693eefbfb7c2b97ba9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-03-11T21:57:37.390 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:37 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:37.282+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:37.391 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:37 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:37.282+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:37.391 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:37 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:37.265+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:37.391 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:37 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:37.265+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:37.739 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:37 smithi110 podman[70765]: 2024-03-11 21:57:37.48763642 +0000 UTC m=+5.985640400 container cleanup b6a67b672a5b39a581aa43e9cc5f45b616cb5819f3599693eefbfb7c2b97ba9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T21:57:37.739 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:37 smithi110 bash[70765]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4 2024-03-11T21:57:38.078 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:37 smithi110 podman[70781]: 2024-03-11 21:57:37.780660974 +0000 UTC m=+0.648767002 container remove b6a67b672a5b39a581aa43e9cc5f45b616cb5819f3599693eefbfb7c2b97ba9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:57:38.078 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:37 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:37.833+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:38.079 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:37 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:37.833+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:38.336 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:38.241+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:38.336 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:38.241+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:38.337 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:38.260+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:38.337 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:38.260+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:38.337 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:38 smithi110 podman[70885]: 2024-03-11 21:57:37.987703287 +0000 UTC m=+0.018221155 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:38.337 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:38 smithi110 podman[70885]: 2024-03-11 21:57:38.098217341 +0000 UTC m=+0.128735206 container create f7acfdefd91b854e5cbc3cc83c7ab3d82e310470084437dc460962acbc2c708d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:57:38.664 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:38 smithi110 podman[70885]: 2024-03-11 21:57:38.377456949 +0000 UTC m=+0.407974827 container init f7acfdefd91b854e5cbc3cc83c7ab3d82e310470084437dc460962acbc2c708d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:57:38.665 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:38 smithi110 podman[70885]: 2024-03-11 21:57:38.380972003 +0000 UTC m=+0.411489879 container start f7acfdefd91b854e5cbc3cc83c7ab3d82e310470084437dc460962acbc2c708d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397) 2024-03-11T21:57:38.665 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:38 smithi110 podman[70885]: 2024-03-11 21:57:38.475806596 +0000 UTC m=+0.506324467 container attach f7acfdefd91b854e5cbc3cc83c7ab3d82e310470084437dc460962acbc2c708d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4-deactivate, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:57:38.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:38.810+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:38.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:38 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:38.810+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:38.989 INFO:journalctl@ceph.osd.4.smithi110.stdout:Mar 11 21:57:38 smithi110 podman[70885]: 2024-03-11 21:57:38.671318124 +0000 UTC m=+0.701835996 container died f7acfdefd91b854e5cbc3cc83c7ab3d82e310470084437dc460962acbc2c708d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-11T21:57:39.298 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:39.234+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:39.298 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:39.234+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:39.298 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:39.254+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:39.298 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:39.254+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:39.328 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T21:57:39.372 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:57:39.373 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T21:57:39.373 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T21:57:39.373 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.5 2024-03-11T21:57:39.701 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:39 smithi110 systemd[1]: Stopping Ceph osd.5 for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:57:39.702 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:39.707+0000 7f164eb8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:57:39.989 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:39.707+0000 7f164eb8d700 -1 osd.5 72 *** Got signal Terminated *** 2024-03-11T21:57:39.989 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:39.707+0000 7f164eb8d700 -1 osd.5 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:57:39.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:39.846+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:39.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:39 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:39.846+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:40.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:40 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:40.237+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:40.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:40 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:40.237+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:40.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:40 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:40.255+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:40.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:40 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:40.255+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:41.223 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:40 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:40.836+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:41.223 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:40 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:40.836+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:41.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:41 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:41.299+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:41.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:41 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:41.299+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:41.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:41 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:41.229+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:41.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:41 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:41.229+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:42.225 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:41 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:41.804+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:42.225 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:41 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:41.804+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:42.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:42.258+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:42.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:42.258+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:42.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:42.231+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:42.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:42.231+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:43.234 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:42.779+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:43.234 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:42 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:42.779+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:43.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:43 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:43.248+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:43.489 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:43 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:43.248+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:43.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:43 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:43.240+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:43.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:43 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:43.240+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:44.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:43 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:43.750+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:44.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:43 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:43.750+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:44.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:43 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:43.750+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:17.395189+0000 front 2024-03-11T21:57:17.395207+0000 (oldest deadline 2024-03-11T21:57:43.295162+0000) 2024-03-11T21:57:44.739 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:44 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:44.278+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.675489+0000 front 2024-03-11T21:57:01.675569+0000 (oldest deadline 2024-03-11T21:57:26.375342+0000) 2024-03-11T21:57:44.739 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:44 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5[43279]: 2024-03-11T21:57:44.278+0000 7f1647179700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.776656+0000 front 2024-03-11T21:57:09.776645+0000 (oldest deadline 2024-03-11T21:57:34.476650+0000) 2024-03-11T21:57:44.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:44 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:44.280+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:44.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:44 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:44.280+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:44.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:44 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:44.280+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:45.239 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:45 smithi110 podman[70990]: 2024-03-11 21:57:45.020437927 +0000 UTC m=+5.419005291 container died 7b2c617469b3dd9d3a126c3edc6438c89e00db536dd8d273fe7f851fc04f9343 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T21:57:45.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:44 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:44.793+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:45.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:44 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:44.793+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:45.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:44 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:44.793+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:17.395189+0000 front 2024-03-11T21:57:17.395207+0000 (oldest deadline 2024-03-11T21:57:43.295162+0000) 2024-03-11T21:57:45.569 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:45 smithi110 podman[70990]: 2024-03-11 21:57:45.349444876 +0000 UTC m=+5.748012240 container cleanup 7b2c617469b3dd9d3a126c3edc6438c89e00db536dd8d273fe7f851fc04f9343 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T21:57:45.569 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:45 smithi110 bash[70990]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5 2024-03-11T21:57:45.569 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:45 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:45.302+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:45.570 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:45 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:45.302+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:45.570 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:45 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:45.302+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:45.570 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:45 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:45.302+0000 7fef09f79700 -1 osd.7 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24406.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72) 2024-03-11T21:57:45.821 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:45 smithi110 podman[71008]: 2024-03-11 21:57:45.575941315 +0000 UTC m=+0.551230071 container remove 7b2c617469b3dd9d3a126c3edc6438c89e00db536dd8d273fe7f851fc04f9343 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T21:57:46.119 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:45 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:45.826+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:46.119 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:45 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:45.826+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:46.119 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:45 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:45.826+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:17.395189+0000 front 2024-03-11T21:57:17.395207+0000 (oldest deadline 2024-03-11T21:57:43.295162+0000) 2024-03-11T21:57:46.119 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:45 smithi110 podman[71103]: 2024-03-11 21:57:45.872907405 +0000 UTC m=+0.018920342 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:46.120 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:46 smithi110 podman[71103]: 2024-03-11 21:57:46.010225179 +0000 UTC m=+0.156238097 container create c44140d21a286fbcf8724db1a30e4f02dcdb4d3f2bb0da0acc9913a90ed4463e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T21:57:46.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:46 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:46.263+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:46.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:46 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:46.263+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:46.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:46 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:46.263+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:46.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:46 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:46.264+0000 7fef09f79700 -1 osd.7 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24406.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72) 2024-03-11T21:57:46.490 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:46 smithi110 podman[71103]: 2024-03-11 21:57:46.204181414 +0000 UTC m=+0.350194336 container init c44140d21a286fbcf8724db1a30e4f02dcdb4d3f2bb0da0acc9913a90ed4463e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:57:46.490 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:46 smithi110 podman[71103]: 2024-03-11 21:57:46.207232096 +0000 UTC m=+0.353245022 container start c44140d21a286fbcf8724db1a30e4f02dcdb4d3f2bb0da0acc9913a90ed4463e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:57:46.490 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:46 smithi110 podman[71103]: 2024-03-11 21:57:46.295860072 +0000 UTC m=+0.441872991 container attach c44140d21a286fbcf8724db1a30e4f02dcdb4d3f2bb0da0acc9913a90ed4463e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-11T21:57:46.800 INFO:journalctl@ceph.osd.5.smithi110.stdout:Mar 11 21:57:46 smithi110 podman[71103]: 2024-03-11 21:57:46.508340822 +0000 UTC m=+0.654353753 container died c44140d21a286fbcf8724db1a30e4f02dcdb4d3f2bb0da0acc9913a90ed4463e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T21:57:47.160 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:46 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:46.807+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:47.160 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:46 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:46.807+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:47.160 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:46 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:46.807+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:17.395189+0000 front 2024-03-11T21:57:17.395207+0000 (oldest deadline 2024-03-11T21:57:43.295162+0000) 2024-03-11T21:57:47.179 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T21:57:47.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:57:47.221 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T21:57:47.221 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T21:57:47.221 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.6 2024-03-11T21:57:47.488 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:47 smithi110 systemd[1]: Stopping Ceph osd.6 for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:57:47.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:47.228+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:47.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:47.228+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:47.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:47.228+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:47.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:47.228+0000 7fef09f79700 -1 osd.7 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24406.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72) 2024-03-11T21:57:47.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:47.586+0000 7f889fa72700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:57:47.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:47.586+0000 7f889fa72700 -1 osd.6 72 *** Got signal Terminated *** 2024-03-11T21:57:47.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:47.586+0000 7f889fa72700 -1 osd.6 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:57:47.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:47.793+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:47.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:47.793+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:47.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:47 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:47.793+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:17.395189+0000 front 2024-03-11T21:57:17.395207+0000 (oldest deadline 2024-03-11T21:57:43.295162+0000) 2024-03-11T21:57:48.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:48 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:48.186+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:48.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:48 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:48.186+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:48.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:48 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:48.186+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:48.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:48 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:48.187+0000 7fef09f79700 -1 osd.7 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24406.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72) 2024-03-11T21:57:49.151 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:48 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:48.764+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:49.151 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:48 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:48.764+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:49.151 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:48 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:48.764+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:17.395189+0000 front 2024-03-11T21:57:17.395207+0000 (oldest deadline 2024-03-11T21:57:43.295162+0000) 2024-03-11T21:57:49.406 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:49 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:49.157+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:49.406 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:49 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:49.157+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:49.406 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:49 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:49.157+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:49.406 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:49 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:49.157+0000 7fef09f79700 -1 osd.7 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24406.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72) 2024-03-11T21:57:50.113 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:49 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:49.780+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:50.113 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:49 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:49.780+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:50.113 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:49 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:49.780+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:17.395189+0000 front 2024-03-11T21:57:17.395207+0000 (oldest deadline 2024-03-11T21:57:43.295162+0000) 2024-03-11T21:57:50.113 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:49 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:49.780+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:23.295600+0000 front 2024-03-11T21:57:23.295584+0000 (oldest deadline 2024-03-11T21:57:49.195507+0000) 2024-03-11T21:57:50.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:50 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:50.119+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:50.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:50 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:50.119+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:50.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:50 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:50.119+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:50.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:50 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:50.119+0000 7fef09f79700 -1 osd.7 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24406.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72) 2024-03-11T21:57:51.064 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:50 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:50.805+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:51.065 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:50 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:50.805+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:51.065 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:50 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:50.805+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:17.395189+0000 front 2024-03-11T21:57:17.395207+0000 (oldest deadline 2024-03-11T21:57:43.295162+0000) 2024-03-11T21:57:51.065 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:50 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:50.805+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:23.295600+0000 front 2024-03-11T21:57:23.295584+0000 (oldest deadline 2024-03-11T21:57:49.195507+0000) 2024-03-11T21:57:51.488 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:51 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:51.070+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:51.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:51 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:51.070+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:51.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:51 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:51.070+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:52.239 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:52 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:52.095+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:52.239 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:52 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:52.095+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:52.239 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:52 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:52.095+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:52.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:51 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:51.854+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.693075+0000 front 2024-03-11T21:57:01.693219+0000 (oldest deadline 2024-03-11T21:57:26.393046+0000) 2024-03-11T21:57:52.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:51 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:51.854+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:11.594307+0000 front 2024-03-11T21:57:11.594241+0000 (oldest deadline 2024-03-11T21:57:35.694143+0000) 2024-03-11T21:57:52.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:51 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:51.854+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:17.395189+0000 front 2024-03-11T21:57:17.395207+0000 (oldest deadline 2024-03-11T21:57:43.295162+0000) 2024-03-11T21:57:52.239 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:51 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6[51190]: 2024-03-11T21:57:51.854+0000 7f889805e700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:23.295600+0000 front 2024-03-11T21:57:23.295584+0000 (oldest deadline 2024-03-11T21:57:49.195507+0000) 2024-03-11T21:57:53.280 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:53 smithi110 podman[71208]: 2024-03-11 21:57:53.017520629 +0000 UTC m=+5.562985946 container died 27dd92278eeda6f2cbbd7438273990530ebf08085d54f1acc78d4afa0cb36dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-11T21:57:53.280 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:53 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:53.090+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:53.281 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:53 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:53.090+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:53.281 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:53 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:53.090+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:53.631 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:53 smithi110 podman[71208]: 2024-03-11 21:57:53.453026928 +0000 UTC m=+5.998492281 container cleanup 27dd92278eeda6f2cbbd7438273990530ebf08085d54f1acc78d4afa0cb36dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T21:57:53.631 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:53 smithi110 bash[71208]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6 2024-03-11T21:57:53.989 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:53 smithi110 podman[71224]: 2024-03-11 21:57:53.637664167 +0000 UTC m=+0.620126217 container remove 27dd92278eeda6f2cbbd7438273990530ebf08085d54f1acc78d4afa0cb36dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6, ceph=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T21:57:54.325 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:54 smithi110 podman[71319]: 2024-03-11 21:57:53.967675935 +0000 UTC m=+0.018151028 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:54.326 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:54 smithi110 podman[71319]: 2024-03-11 21:57:54.105348357 +0000 UTC m=+0.155823446 container create 37ee4b28971aca40d1257b650349b1b3a189bbb30f1e1a632dbee2a7a4f68f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T21:57:54.326 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:54 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:54.125+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:54.326 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:54 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:54.125+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:54.326 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:54 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:54.125+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:54.326 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:54 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:54.125+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:27.379425+0000 front 2024-03-11T21:57:27.379408+0000 (oldest deadline 2024-03-11T21:57:53.279316+0000) 2024-03-11T21:57:54.617 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:54 smithi110 podman[71319]: 2024-03-11 21:57:54.332591813 +0000 UTC m=+0.383066909 container init 37ee4b28971aca40d1257b650349b1b3a189bbb30f1e1a632dbee2a7a4f68f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:57:54.617 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:54 smithi110 podman[71319]: 2024-03-11 21:57:54.335909294 +0000 UTC m=+0.386384385 container start 37ee4b28971aca40d1257b650349b1b3a189bbb30f1e1a632dbee2a7a4f68f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T21:57:54.617 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:54 smithi110 podman[71319]: 2024-03-11 21:57:54.432710642 +0000 UTC m=+0.483185733 container attach 37ee4b28971aca40d1257b650349b1b3a189bbb30f1e1a632dbee2a7a4f68f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T21:57:54.904 INFO:journalctl@ceph.osd.6.smithi110.stdout:Mar 11 21:57:54 smithi110 podman[71319]: 2024-03-11 21:57:54.623668697 +0000 UTC m=+0.674143792 container died 37ee4b28971aca40d1257b650349b1b3a189bbb30f1e1a632dbee2a7a4f68f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T21:57:55.238 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:55 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:55.076+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:55.239 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:55 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:55.076+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:55.239 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:55 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:55.076+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:55.239 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:55 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:55.076+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:27.379425+0000 front 2024-03-11T21:57:27.379408+0000 (oldest deadline 2024-03-11T21:57:53.279316+0000) 2024-03-11T21:57:55.263 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T21:57:55.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:57:55.303 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T21:57:55.304 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T21:57:55.304 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.7 2024-03-11T21:57:55.609 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:55 smithi110 systemd[1]: Stopping Ceph osd.7 for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:57:55.988 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:55 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:55.613+0000 7fef0d985700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:57:55.989 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:55 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:55.613+0000 7fef0d985700 -1 osd.7 72 *** Got signal Terminated *** 2024-03-11T21:57:55.989 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:55 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:55.613+0000 7fef0d985700 -1 osd.7 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:57:56.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:56.042+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:56.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:56.042+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:56.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:56.042+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:56.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:56 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:56.042+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:27.379425+0000 front 2024-03-11T21:57:27.379408+0000 (oldest deadline 2024-03-11T21:57:53.279316+0000) 2024-03-11T21:57:57.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:57 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:57.032+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:57.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:57 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:57.032+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:57.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:57 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:57.032+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:57.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:57 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:57.032+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:27.379425+0000 front 2024-03-11T21:57:27.379408+0000 (oldest deadline 2024-03-11T21:57:53.279316+0000) 2024-03-11T21:57:58.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:58 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:58.047+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:58.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:58 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:58.047+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:58.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:58 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:58.047+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:58.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:58 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:58.047+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:27.379425+0000 front 2024-03-11T21:57:27.379408+0000 (oldest deadline 2024-03-11T21:57:53.279316+0000) 2024-03-11T21:57:58.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:58 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:58.047+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.110:6804 osd.4 since back 2024-03-11T21:57:33.279709+0000 front 2024-03-11T21:57:33.279808+0000 (oldest deadline 2024-03-11T21:57:57.379715+0000) 2024-03-11T21:57:59.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:59 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:59.008+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:57:59.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:59 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:59.008+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:57:59.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:59 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:59.008+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:57:59.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:59 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:59.008+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:27.379425+0000 front 2024-03-11T21:57:27.379408+0000 (oldest deadline 2024-03-11T21:57:53.279316+0000) 2024-03-11T21:57:59.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:57:59 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:57:59.008+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.110:6804 osd.4 since back 2024-03-11T21:57:33.279709+0000 front 2024-03-11T21:57:33.279808+0000 (oldest deadline 2024-03-11T21:57:57.379715+0000) 2024-03-11T21:58:00.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:00 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:58:00.057+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-11T21:57:01.676440+0000 front 2024-03-11T21:57:01.676356+0000 (oldest deadline 2024-03-11T21:57:26.976298+0000) 2024-03-11T21:58:00.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:00 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:58:00.057+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-11T21:57:09.277153+0000 front 2024-03-11T21:57:09.277163+0000 (oldest deadline 2024-03-11T21:57:34.577096+0000) 2024-03-11T21:58:00.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:00 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:58:00.057+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-11T21:57:19.178556+0000 front 2024-03-11T21:57:19.178729+0000 (oldest deadline 2024-03-11T21:57:43.878587+0000) 2024-03-11T21:58:00.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:00 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:58:00.057+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-03-11T21:57:27.379425+0000 front 2024-03-11T21:57:27.379408+0000 (oldest deadline 2024-03-11T21:57:53.279316+0000) 2024-03-11T21:58:00.489 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:00 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7[59480]: 2024-03-11T21:58:00.057+0000 7fef09f79700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.110:6804 osd.4 since back 2024-03-11T21:57:33.279709+0000 front 2024-03-11T21:57:33.279808+0000 (oldest deadline 2024-03-11T21:57:57.379715+0000) 2024-03-11T21:58:01.375 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:01 smithi110 podman[71423]: 2024-03-11 21:58:01.075056252 +0000 UTC m=+5.544986836 container died 9f922576ebac48db16d89583088ea102d03946c341c8a793cd1364f199c64464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-11T21:58:01.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:01 smithi110 podman[71423]: 2024-03-11 21:58:01.564486526 +0000 UTC m=+6.034417130 container cleanup 9f922576ebac48db16d89583088ea102d03946c341c8a793cd1364f199c64464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:58:01.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:01 smithi110 bash[71423]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7 2024-03-11T21:58:02.083 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:01 smithi110 podman[71440]: 2024-03-11 21:58:01.777123435 +0000 UTC m=+0.699970238 container remove 9f922576ebac48db16d89583088ea102d03946c341c8a793cd1364f199c64464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T21:58:02.404 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:02 smithi110 podman[71532]: 2024-03-11 21:58:01.992492528 +0000 UTC m=+0.018030511 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:58:02.404 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:02 smithi110 podman[71532]: 2024-03-11 21:58:02.116825781 +0000 UTC m=+0.142363802 container create 3704c8cdf372ab8176c94fb2707cf05cfe9544f6514887ffb21bffb940763446 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-11T21:58:02.404 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:02 smithi110 podman[71532]: 2024-03-11 21:58:02.33576485 +0000 UTC m=+0.361302828 container init 3704c8cdf372ab8176c94fb2707cf05cfe9544f6514887ffb21bffb940763446 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T21:58:02.404 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:02 smithi110 podman[71532]: 2024-03-11 21:58:02.33896743 +0000 UTC m=+0.364505409 container start 3704c8cdf372ab8176c94fb2707cf05cfe9544f6514887ffb21bffb940763446 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T21:58:02.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:02 smithi110 podman[71532]: 2024-03-11 21:58:02.410724247 +0000 UTC m=+0.436262236 container attach 3704c8cdf372ab8176c94fb2707cf05cfe9544f6514887ffb21bffb940763446 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-11T21:58:02.739 INFO:journalctl@ceph.osd.7.smithi110.stdout:Mar 11 21:58:02 smithi110 podman[71532]: 2024-03-11 21:58:02.630869094 +0000 UTC m=+0.656407073 container died 3704c8cdf372ab8176c94fb2707cf05cfe9544f6514887ffb21bffb940763446 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397) 2024-03-11T21:58:03.302 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T21:58:03.342 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:58:03.343 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T21:58:03.343 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T21:58:03.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T21:58:03.640 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:03 smithi102 systemd[1]: Stopping Ceph rgw.foo.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:58:03.919 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:03 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a[70222]: 2024-03-11T21:58:03.645+0000 7fdde3f06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:58:03.919 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:03 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a[70222]: 2024-03-11T21:58:03.645+0000 7fddf2776ac0 -1 shutting down 2024-03-11T21:58:13.919 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:13 smithi102 bash[81514]: time="2024-03-11T21:58:13Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T21:58:13.920 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:13 smithi102 podman[81514]: 2024-03-11 21:58:13.661970755 +0000 UTC m=+10.118449074 container died 7543b2522bb338bc367d60cc126d2fe5fb4089917d34e698f4d930df952eaebf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T21:58:14.315 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:14 smithi102 podman[81514]: 2024-03-11 21:58:14.095576361 +0000 UTC m=+10.552054717 container cleanup 7543b2522bb338bc367d60cc126d2fe5fb4089917d34e698f4d930df952eaebf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T21:58:14.315 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:14 smithi102 bash[81514]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T21:58:14.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T21:58:14.589 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:14 smithi102 podman[81543]: 2024-03-11 21:58:14.321629014 +0000 UTC m=+0.651322587 container remove 7543b2522bb338bc367d60cc126d2fe5fb4089917d34e698f4d930df952eaebf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:58:14.589 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:14 smithi102 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T21:58:14.589 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:14 smithi102 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-11T21:58:14.589 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:14 smithi102 systemd[1]: Stopped Ceph rgw.foo.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:58:14.589 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 11 21:58:14 smithi102 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service: Consumed 1.482s CPU time. 2024-03-11T21:58:14.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:58:14.603 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T21:58:14.603 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T21:58:14.604 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T21:58:14.909 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 systemd[1]: Stopping Ceph prometheus.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.915Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T21:58:15.203 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T21:58:15.204 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T21:58:15.204 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a[68205]: ts=2024-03-11T21:58:14.916Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T21:58:15.204 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:14 smithi110 podman[71637]: 2024-03-11 21:58:14.92644963 +0000 UTC m=+0.124496412 container died ae0fd8d4739ea7a7aea61009c0ee713bc70fff17b35626cb9cfa9ffa80332a2f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:58:15.489 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:15 smithi110 podman[71637]: 2024-03-11 21:58:15.408127992 +0000 UTC m=+0.606174771 container cleanup ae0fd8d4739ea7a7aea61009c0ee713bc70fff17b35626cb9cfa9ffa80332a2f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:58:15.489 INFO:journalctl@ceph.prometheus.a.smithi110.stdout:Mar 11 21:58:15 smithi110 bash[71637]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T21:58:15.773 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T21:58:15.811 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:58:15.811 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T21:58:15.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:58:15.993 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:58:17.919 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:58:17 smithi102 systemd[1]: Stopping Ceph alertmanager.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:58:18.278 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:58:17 smithi102 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a[78324]: ts=2024-03-11T21:58:17.983Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:58:18.278 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:58:18 smithi102 podman[82023]: 2024-03-11 21:58:18.004359619 +0000 UTC m=+0.188173072 container died c29d8bfd1c1f8a7d4ad021d7896e47cc6a6bbb637854d77c87ed8705d0cd76a8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:58:18.620 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:58:18 smithi102 podman[82023]: 2024-03-11 21:58:18.375855945 +0000 UTC m=+0.559669390 container cleanup c29d8bfd1c1f8a7d4ad021d7896e47cc6a6bbb637854d77c87ed8705d0cd76a8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:58:18.620 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:58:18 smithi102 bash[82023]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:58:18.919 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:58:18 smithi102 podman[82036]: 2024-03-11 21:58:18.626763109 +0000 UTC m=+0.617623728 container remove c29d8bfd1c1f8a7d4ad021d7896e47cc6a6bbb637854d77c87ed8705d0cd76a8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:58:18.920 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:58:18 smithi102 podman[82036]: 2024-03-11 21:58:18.702434994 +0000 UTC m=+0.693295624 volume remove 45e027222c09c11abb05eb11b05da8a3a926d94b6fe36a60a11b81a58841f7c3 2024-03-11T21:58:18.920 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:58:18 smithi102 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T21:58:18.920 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 11 21:58:18 smithi102 systemd[1]: Stopped Ceph alertmanager.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:58:19.419 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:58:19 smithi102 systemd[1]: Stopping Ceph node-exporter.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:58:19.419 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:58:19 smithi102 podman[82150]: 2024-03-11 21:58:19.27394888 +0000 UTC m=+0.104963917 container died 39b53abb1c4d13b853c10accafb7c9971f305d348986542110aaf7e26da4e2cc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:58:19.774 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:58:19 smithi102 podman[82150]: 2024-03-11 21:58:19.588280256 +0000 UTC m=+0.419295294 container cleanup 39b53abb1c4d13b853c10accafb7c9971f305d348986542110aaf7e26da4e2cc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:58:19.774 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:58:19 smithi102 bash[82150]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T21:58:20.169 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:58:19 smithi102 podman[82164]: 2024-03-11 21:58:19.78100776 +0000 UTC m=+0.511730258 container remove 39b53abb1c4d13b853c10accafb7c9971f305d348986542110aaf7e26da4e2cc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:58:20.169 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:58:19 smithi102 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:58:20.169 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:58:20 smithi102 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T21:58:20.169 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:58:20 smithi102 systemd[1]: Stopped Ceph node-exporter.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:58:20.170 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 11 21:58:20 smithi102 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 1.748s CPU time. 2024-03-11T21:58:20.642 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:58:20.833 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:58:22.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:22 smithi110 systemd[1]: Stopping Ceph iscsi.iscsi.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:58:22.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:22 smithi110 ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a[66618]: debug Shutdown received 2024-03-11T21:58:32.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:32 smithi110 bash[72059]: time="2024-03-11T21:58:32Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-11T21:58:32.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:32 smithi110 podman[72059]: 2024-03-11 21:58:32.426913668 +0000 UTC m=+10.112749084 container died 6e2c30803a349f234eb5d82936be9c3446b28d177c60dd0588e5894997a6a16d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-11T21:58:32.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:32 smithi110 podman[72059]: 2024-03-11 21:58:32.878676231 +0000 UTC m=+10.564511635 container cleanup 6e2c30803a349f234eb5d82936be9c3446b28d177c60dd0588e5894997a6a16d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T21:58:32.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:32 smithi110 bash[72059]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-11T21:58:33.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:33 smithi110 podman[72074]: 2024-03-11 21:58:33.171880306 +0000 UTC m=+0.741866140 container remove 6e2c30803a349f234eb5d82936be9c3446b28d177c60dd0588e5894997a6a16d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T21:58:33.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:33 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T21:58:33.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:33 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-11T21:58:33.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:33 smithi110 systemd[1]: Stopped Ceph iscsi.iscsi.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:58:33.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi110.stdout:Mar 11 21:58:33 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Consumed 1.974s CPU time. 2024-03-11T21:58:34.989 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:58:34 smithi110 systemd[1]: Stopping Ceph grafana.a for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:58:35.290 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:58:35 smithi110 podman[72339]: 2024-03-11 21:58:35.030785428 +0000 UTC m=+0.104238912 container died b9cbc7756b8b7184ed6a8aa4bc11e5150bd26a3cbcf8d4856f555ba66ea644a9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.openshift.expose-services=, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, distribution-scope=public, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, release=239.1651231664, name=ubi8) 2024-03-11T21:58:35.641 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:58:35 smithi110 podman[72339]: 2024-03-11 21:58:35.470693956 +0000 UTC m=+0.544147451 container cleanup b9cbc7756b8b7184ed6a8aa4bc11e5150bd26a3cbcf8d4856f555ba66ea644a9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, architecture=x86_64, io.buildah.version=1.32.0, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-11T21:58:35.641 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:58:35 smithi110 bash[72339]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a 2024-03-11T21:58:35.894 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:58:35 smithi110 podman[72353]: 2024-03-11 21:58:35.646975333 +0000 UTC m=+0.618600500 container remove b9cbc7756b8b7184ed6a8aa4bc11e5150bd26a3cbcf8d4856f555ba66ea644a9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, vcs-type=git, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., version=8.5, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, maintainer=Paul Cuzner ) 2024-03-11T21:58:35.894 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:58:35 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:58:35.895 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:58:35 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T21:58:35.895 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:58:35 smithi110 systemd[1]: Stopped Ceph grafana.a for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:58:35.895 INFO:journalctl@ceph.grafana.a.smithi110.stdout:Mar 11 21:58:35 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@grafana.a.service: Consumed 2.984s CPU time. 2024-03-11T21:58:36.239 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:58:36 smithi110 systemd[1]: Stopping Ceph node-exporter.b for 0126e6b2-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:58:36.676 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:58:36 smithi110 podman[72463]: 2024-03-11 21:58:36.360738419 +0000 UTC m=+0.138704890 container died 56a6f1570c42c3c7c6e1d41aa9569863e035b0817e06c244c6c3c31b02ca569a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:58:36.676 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:58:36 smithi110 podman[72463]: 2024-03-11 21:58:36.68327194 +0000 UTC m=+0.461238417 container cleanup 56a6f1570c42c3c7c6e1d41aa9569863e035b0817e06c244c6c3c31b02ca569a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:58:36.989 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:58:36 smithi110 bash[72463]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T21:58:36.989 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:58:36 smithi110 podman[72477]: 2024-03-11 21:58:36.859247512 +0000 UTC m=+0.503220116 container remove 56a6f1570c42c3c7c6e1d41aa9569863e035b0817e06c244c6c3c31b02ca569a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0126e6b2-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:58:36.989 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:58:36 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:58:37.489 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:58:37 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T21:58:37.489 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:58:37 smithi110 systemd[1]: Stopped Ceph node-exporter.b for 0126e6b2-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:58:37.489 INFO:journalctl@ceph.node-exporter.b.smithi110.stdout:Mar 11 21:58:37 smithi110 systemd[1]: ceph-0126e6b2-dff1-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 1.714s CPU time. 2024-03-11T21:58:37.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:58:37.787 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:58:37.816 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T21:58:37.818 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591936/remote/smithi102/crash 2024-03-11T21:58:37.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:58:37.867 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:58:37.867 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:58:37.868 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591936/remote/smithi110/crash 2024-03-11T21:58:37.869 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:58:37.904 INFO:teuthology.orchestra.run.smithi110.stderr:tar: /var/lib/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:58:37.905 INFO:teuthology.orchestra.run.smithi110.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:58:37.906 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T21:58:37.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-11T21:58:37.942 INFO:tasks.cephadm:Compressing logs... 2024-03-11T21:58:37.942 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:58:37.985 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:58:38.011 INFO:teuthology.orchestra.run.smithi110.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:58:38.012 INFO:teuthology.orchestra.run.smithi110.stderr:: No such file or directory 2024-03-11T21:58:38.012 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T21:58:38.012 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:58:38.012 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T21:58:38.013 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T21:58:38.013 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T21:58:38.013 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.4.log: /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T21:58:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’gzip: No such file or directory -5 2024-03-11T21:58:38.015 INFO:teuthology.orchestra.run.smithi102.stderr: --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:58:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T21:58:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:58:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T21:58:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T21:58:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T21:58:38.015 INFO:teuthology.orchestra.run.smithi102.stderr: 57.7% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T21:58:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T21:58:38.017 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-11T21:58:38.017 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:58:38.017 INFO:teuthology.orchestra.run.smithi110.stderr: 90.0% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:58:38.017 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/tcmu-runner.log: 67.2% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-11T21:58:38.017 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:58:38.018 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:58:38.018 INFO:teuthology.orchestra.run.smithi110.stderr: 86.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:58:38.018 INFO:teuthology.orchestra.run.smithi110.stderr: 81.5% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:58:38.018 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.log: 85.7% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:58:38.018 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mgr.y.log: gzip -5 91.7% --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 93.8% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr: --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr: 88.9% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.log: -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T21:58:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-volume.log: 91.7% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:58:38.022 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mon.b.log: 90.5% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T21:58:38.029 INFO:teuthology.orchestra.run.smithi110.stderr: 90.5% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:58:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.5% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:58:38.138 INFO:teuthology.orchestra.run.smithi110.stderr: 92.0% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T21:58:38.155 INFO:teuthology.orchestra.run.smithi102.stderr: 89.0% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T21:58:38.162 INFO:teuthology.orchestra.run.smithi102.stderr: 92.4% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T21:58:38.216 INFO:teuthology.orchestra.run.smithi110.stderr: 93.4% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T21:58:38.216 INFO:teuthology.orchestra.run.smithi110.stderr: 93.0% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T21:58:38.217 INFO:teuthology.orchestra.run.smithi110.stderr: 93.1% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T21:58:38.234 INFO:teuthology.orchestra.run.smithi110.stderr: 93.1% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T21:58:38.235 INFO:teuthology.orchestra.run.smithi102.stderr: 92.9% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T21:58:38.236 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-11T21:58:38.236 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m0.237s 2024-03-11T21:58:38.236 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m0.965s 2024-03-11T21:58:38.237 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m0.049s 2024-03-11T21:58:38.256 INFO:teuthology.orchestra.run.smithi102.stderr: 93.5% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T21:58:38.264 INFO:teuthology.orchestra.run.smithi102.stderr: 93.5% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T21:58:38.274 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T21:58:38.327 INFO:teuthology.orchestra.run.smithi102.stderr: 91.1% -- replaced with /var/log/ceph/0126e6b2-dff1-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T21:58:38.329 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-11T21:58:38.329 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m0.330s 2024-03-11T21:58:38.329 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m1.542s 2024-03-11T21:58:38.329 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m0.076s 2024-03-11T21:58:38.329 INFO:tasks.cephadm:Archiving logs... 2024-03-11T21:58:38.330 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591936/remote/smithi102/log 2024-03-11T21:58:38.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:58:38.605 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591936/remote/smithi110/log 2024-03-11T21:58:38.606 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:58:38.772 INFO:tasks.cephadm:Removing cluster... 2024-03-11T21:58:38.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 --force 2024-03-11T21:58:38.964 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:58:39.091 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0126e6b2-dff1-11ee-95c8-87774f69a715 --force 2024-03-11T21:58:39.272 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 0126e6b2-dff1-11ee-95c8-87774f69a715 2024-03-11T21:58:39.385 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T21:58:39.386 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:58:39.401 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:58:39.416 INFO:tasks.cephadm:Teardown complete 2024-03-11T21:58:39.417 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-11T21:58:39.427 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_1... 2024-03-11T21:58:39.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_1 2024-03-11T21:58:39.597 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T21:58:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:58:39.599 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_2... 2024-03-11T21:58:39.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_2 2024-03-11T21:58:39.737 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T21:58:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:58:39.739 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_3... 2024-03-11T21:58:39.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_3 2024-03-11T21:58:39.873 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T21:58:39.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:58:39.875 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_4... 2024-03-11T21:58:39.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_4 2024-03-11T21:58:40.027 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T21:58:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:58:40.029 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-11T21:58:40.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/scratch_devs 2024-03-11T21:58:40.055 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi110:/dev/vg_nvme/lv_1... 2024-03-11T21:58:40.055 DEBUG:teuthology.orchestra.run.smithi110:> sudo nvme disconnect -n lv_1 2024-03-11T21:58:40.191 INFO:teuthology.orchestra.run.smithi110.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T21:58:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:58:40.193 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi110:/dev/vg_nvme/lv_2... 2024-03-11T21:58:40.193 DEBUG:teuthology.orchestra.run.smithi110:> sudo nvme disconnect -n lv_2 2024-03-11T21:58:40.355 INFO:teuthology.orchestra.run.smithi110.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T21:58:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:58:40.357 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi110:/dev/vg_nvme/lv_3... 2024-03-11T21:58:40.357 DEBUG:teuthology.orchestra.run.smithi110:> sudo nvme disconnect -n lv_3 2024-03-12T09:39:51.088 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-12T09:39:51.119 DEBUG:teuthology.task.console_log:Killing console logger for smithi102 2024-03-12T09:39:51.120 DEBUG:teuthology.task.console_log:Killing console logger for smithi110 2024-03-12T09:39:51.120 DEBUG:teuthology.exit:Finished running handlers