2024-09-17T08:51:42.983 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T08:51:43.000 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T08:51:43.077 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908264 branch: reef description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} email: ceph-qa@ceph.com first_in_suite: false job_id: '7908264' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: global: mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef sha1: 1cbdc9211df088cccbd94423c7cf3a2b85fe432b owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 5234 sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 sleep_before_teardown: 0 subset: 46/64 suite: orch suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1cbdc9211df088cccbd94423c7cf3a2b85fe432b targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGT0WJksHTzdfbBTWuRZ9ZF4jnWjiOWAUvieTIFGJO9Vc0hg7kErM8gD/qh3m0Hph8gL/9TBldtd7XcoARqo7o= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSiOf/jBWm5xtg7vES+ppbXwtgH7spA6XlX32b3PmM/wrsn4xAFbI6Oc82CDTuFoE1ONeyULFiMA4icRZ2LOoA= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - "set -ex\nfor f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x\n\ do\n echo \"rotating key for $f\"\n K=$(ceph auth get-key $f)\n NK=\"\ $K\"\n ceph orch daemon rotate-key $f\n while [ \"$K\" == \"$NK\" ]; do\n\ \ sleep 5\n NK=$(ceph auth get-key $f)\n done\ndone\n" teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_22:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T08:51:43.077 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa; will attempt to use it 2024-09-17T08:51:43.078 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa/tasks 2024-09-17T08:51:43.078 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T08:51:43.079 INFO:teuthology.task.internal:Checking packages... 2024-09-17T08:51:43.102 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9' 2024-09-17T08:51:43.102 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T08:51:43.102 INFO:teuthology.packaging:ref: None 2024-09-17T08:51:43.103 INFO:teuthology.packaging:tag: None 2024-09-17T08:51:43.103 INFO:teuthology.packaging:branch: reef 2024-09-17T08:51:43.103 INFO:teuthology.packaging:sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T08:51:43.103 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-17T08:51:43.353 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-855.g19e2a2be 2024-09-17T08:51:43.355 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T08:51:43.361 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T08:51:43.361 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T08:51:43.380 INFO:teuthology.task.internal:Saving configuration 2024-09-17T08:51:43.391 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T08:51:43.398 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T08:51:43.425 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908264', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 08:46:43.605727', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGT0WJksHTzdfbBTWuRZ9ZF4jnWjiOWAUvieTIFGJO9Vc0hg7kErM8gD/qh3m0Hph8gL/9TBldtd7XcoARqo7o='} 2024-09-17T08:51:43.445 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908264', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 08:46:43.607267', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSiOf/jBWm5xtg7vES+ppbXwtgH7spA6XlX32b3PmM/wrsn4xAFbI6Oc82CDTuFoE1ONeyULFiMA4icRZ2LOoA='} 2024-09-17T08:51:43.445 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T08:51:43.451 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-17T08:51:43.451 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-17T08:51:43.451 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T08:51:43.520 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3fe0b3bac0>, signals=[15]) 2024-09-17T08:51:43.520 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T08:51:43.526 INFO:teuthology.task.internal:Opening connections... 2024-09-17T08:51:43.527 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-09-17T08:51:43.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T08:51:43.611 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T08:51:43.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T08:51:43.685 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T08:51:43.692 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-09-17T08:51:43.711 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-09-17T08:51:43.712 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-09-17T08:51:43.770 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-09-17T08:51:43.770 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-09-17T08:51:43.770 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-09-17T08:51:43.770 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-09-17T08:51:43.770 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-09-17T08:51:43.771 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-09-17T08:51:43.771 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T08:51:43.771 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-09-17T08:51:43.771 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-09-17T08:51:43.771 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T08:51:43.771 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-09-17T08:51:43.771 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T08:51:43.771 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T08:51:43.771 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T08:51:43.772 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-09-17T08:51:43.797 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-09-17T08:51:43.815 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-09-17T08:51:43.815 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-09-17T08:51:43.873 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-09-17T08:51:43.873 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-09-17T08:51:43.874 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-09-17T08:51:43.874 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-09-17T08:51:43.874 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-09-17T08:51:43.874 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-09-17T08:51:43.874 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T08:51:43.874 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-09-17T08:51:43.874 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-09-17T08:51:43.875 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T08:51:43.875 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-09-17T08:51:43.875 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T08:51:43.875 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T08:51:43.875 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T08:51:43.876 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-09-17T08:51:43.897 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T08:51:43.906 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T08:51:43.933 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T08:51:43.933 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-09-17T08:51:43.937 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-09-17T08:51:43.955 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T08:51:43.962 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T08:51:43.962 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-09-17T08:51:43.994 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-09-17T08:51:44.028 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T08:51:44.081 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T08:51:44.097 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T08:51:44.097 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T08:51:44.097 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T08:51:44.098 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T08:51:44.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T08:51:44.114 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-09-17T08:51:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T08:51:44.116 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-09-17T08:51:44.172 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-503.el9.x86_64 2024-09-17T08:51:44.172 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-503.el9.x86_64 2024-09-17T08:51:44.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-09-17T08:51:44.173 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-503.el9.x86_64 2024-09-17T08:51:44.173 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-503.el9.x86_64 2024-09-17T08:51:44.174 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-09-17T08:51:47.732 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.3 MB 00:01 2024-09-17T08:51:47.840 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.3 MB 00:01 2024-09-17T08:51:49.508 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-17T08:51:54.850 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-09-17T08:51:57.060 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-17T08:51:58.163 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-17T08:52:03.569 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-09-17T08:52:04.073 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 20 MB 00:15 2024-09-17T08:52:05.450 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T08:52:05.450 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T08:52:05.544 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T08:52:05.557 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:52:05.557 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-09-17T08:52:05.557 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:52:05.557 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T08:52:05.557 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T08:52:05.557 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T08:52:05.557 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 79 M 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 126 M 2024-09-17T08:52:05.558 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T08:52:05.978 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 123 kB/s | 36 kB 00:00 2024-09-17T08:52:09.448 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.5 MB 00:01 2024-09-17T08:52:09.877 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 3.7 MB/s | 16 MB 00:04 2024-09-17T08:52:10.677 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 6.9 MB/s | 35 MB 00:04 2024-09-17T08:52:11.198 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 19 kB 00:00 2024-09-17T08:52:11.961 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-17T08:52:17.383 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-17T08:52:19.298 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T08:52:19.299 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T08:52:19.381 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T08:52:19.391 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:52:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T08:52:19.391 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:52:19.391 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T08:52:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T08:52:19.391 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T08:52:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T08:52:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T08:52:19.392 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T08:52:19.392 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:52:19.392 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T08:52:19.392 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:52:19.392 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-09-17T08:52:19.392 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:52:19.393 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 79 M 2024-09-17T08:52:19.393 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 126 M 2024-09-17T08:52:19.393 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T08:52:19.576 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 429 kB/s | 36 kB 00:00 2024-09-17T08:52:21.259 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 1.9 MB/s | 29 MB 00:15 2024-09-17T08:52:21.260 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T08:52:21.260 INFO:teuthology.orchestra.run.smithi079.stdout:Total 5.0 MB/s | 79 MB 00:15 2024-09-17T08:52:21.943 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T08:52:22.305 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T08:52:22.306 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T08:52:23.279 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T08:52:23.280 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T08:52:23.426 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 4.0 MB/s | 16 MB 00:03 2024-09-17T08:52:24.127 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 6.3 MB/s | 29 MB 00:04 2024-09-17T08:52:25.693 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T08:52:26.343 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T08:52:26.435 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T08:52:28.293 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 3.9 MB/s | 35 MB 00:08 2024-09-17T08:52:28.293 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T08:52:28.293 INFO:teuthology.orchestra.run.smithi173.stdout:Total 8.9 MB/s | 79 MB 00:08 2024-09-17T08:52:28.979 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T08:52:29.176 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T08:52:29.328 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T08:52:29.328 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T08:52:29.337 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T08:52:30.262 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T08:52:30.262 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T08:52:32.670 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T08:52:33.392 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T08:52:33.476 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T08:52:34.177 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T08:52:34.287 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:52:36.250 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T08:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T08:52:39.110 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:52:41.242 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:52:46.097 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:53:17.335 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:53:17.374 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:53:17.719 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:53:17.719 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T08:53:17.719 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T08:53:17.719 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T08:53:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:53:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:53:18.556 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T08:53:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T08:53:18.557 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T08:53:18.557 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T08:53:18.557 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T08:53:18.557 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:53:18.557 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T08:53:18.942 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T08:53:19.032 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T08:53:19.032 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T08:53:19.032 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T08:53:19.033 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T08:53:19.033 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T08:53:19.033 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T08:53:19.033 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T08:53:19.033 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T08:53:19.092 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T08:53:19.093 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T08:53:19.093 DEBUG:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2-tools 2024-09-17T08:53:19.127 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2-tools 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.06 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 82.el9 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Group : Unspecified 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 8274670 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T08:53:19.128 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : builder@centos.org 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Support tools for GRUB. 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:53:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T08:53:19.130 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T08:53:19.130 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 5.14.0-508.el9.x86_64 2024-09-17T08:53:19.130 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T08:53:19.767 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-09-17T08:53:21.095 INFO:teuthology.orchestra.run.smithi079.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T08:53:21.127 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-09-17T08:53:21.129 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-09-17T08:53:21.145 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.15n74Tsqmv 2024-09-17T08:53:21.145 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.15n74Tsqmv 2024-09-17T08:53:21.217 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.15n74Tsqmv 2024-09-17T08:53:21.360 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.15n74Tsqmv is 6KB 2024-09-17T08:53:21.409 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.15n74Tsqmv 2024-09-17T08:53:21.424 DEBUG:teuthology.orchestra.run.smithi079:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T08:53:21.495 INFO:teuthology.orchestra.run.smithi079.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T08:53:21.495 INFO:teuthology.orchestra.run.smithi079.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T08:53:21.495 INFO:teuthology.orchestra.run.smithi079.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T08:53:21.495 INFO:teuthology.orchestra.run.smithi079.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T08:53:21.496 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T08:53:21.651 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-09-17T08:53:24.505 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:53:24.561 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:53:24.917 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:53:24.917 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T08:53:24.918 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T08:53:24.918 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T08:53:25.786 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T08:53:25.786 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:53:25.786 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T08:53:25.786 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T08:53:25.787 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T08:53:25.787 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T08:53:25.787 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T08:53:25.787 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:53:25.787 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T08:53:26.135 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T08:53:26.223 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T08:53:26.223 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T08:53:26.223 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T08:53:26.223 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T08:53:26.223 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T08:53:26.223 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T08:53:26.224 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T08:53:26.224 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T08:53:26.283 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T08:53:26.283 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T08:53:26.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-09-17T08:53:26.315 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-09-17T08:53:26.315 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-09-17T08:53:26.315 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-09-17T08:53:26.315 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 82.el9 2024-09-17T08:53:26.315 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-09-17T08:53:26.315 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T08:53:26.315 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-09-17T08:53:26.315 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8274670 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:53:26.316 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T08:53:26.317 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T08:53:26.317 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T08:53:26.317 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-09-17T08:53:26.317 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:53:26.317 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T08:53:26.318 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T08:53:26.318 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-508.el9.x86_64 2024-09-17T08:53:26.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T08:53:26.898 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-09-17T08:53:28.211 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T08:53:28.243 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-09-17T08:53:28.245 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-09-17T08:53:28.261 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.mfb2kc0lnd 2024-09-17T08:53:28.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mfb2kc0lnd 2024-09-17T08:53:28.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.mfb2kc0lnd 2024-09-17T08:53:28.472 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.mfb2kc0lnd is 6KB 2024-09-17T08:53:28.492 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.mfb2kc0lnd 2024-09-17T08:53:28.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T08:53:28.579 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T08:53:28.580 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T08:53:28.580 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T08:53:28.580 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T08:53:28.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T08:53:28.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-09-17T08:53:51.676 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T08:53:51.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-17T08:53:51.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T08:53:58.743 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T08:53:58.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T08:53:58.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T08:54:33.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-09-17T08:54:42.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T08:54:42.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T08:54:51.689 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T08:55:00.673 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-09-17T08:55:00.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-17T08:55:00.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T08:55:01.138 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-09-17T08:55:01.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-17T08:55:01.513 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T08:55:01.513 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-09-17T08:55:01.531 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-508.el9.x86_64 2024-09-17T08:55:01.531 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T08:55:01.531 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T08:55:01.531 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T08:55:02.532 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T08:55:02.532 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T08:55:02.641 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-09-17T08:55:02.677 DEBUG:teuthology.parallel:result is None 2024-09-17T08:55:12.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T08:55:12.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T08:55:13.006 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-09-17T08:55:13.354 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T08:55:13.354 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T08:55:13.354 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-09-17T08:55:13.373 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-508.el9.x86_64 2024-09-17T08:55:13.373 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T08:55:13.373 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T08:55:13.374 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T08:55:14.374 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T08:55:14.374 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T08:55:14.463 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-09-17T08:55:14.484 DEBUG:teuthology.parallel:result is None 2024-09-17T08:55:14.484 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T08:55:14.494 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T08:55:14.495 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T08:55:14.498 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T08:55:14.515 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T08:55:14.521 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T08:55:14.530 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T08:55:14.531 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T08:55:14.558 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T08:55:14.674 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T08:55:14.680 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T08:55:14.680 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T08:55:14.683 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T08:55:14.711 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T08:55:14.713 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T08:55:14.723 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T08:55:14.723 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T08:55:14.725 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T08:55:14.732 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T08:55:14.733 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T08:55:14.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T08:55:14.793 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T08:55:14.801 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T08:55:14.801 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T08:55:14.835 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T08:55:14.849 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T08:55:14.933 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T08:55:14.982 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T08:55:15.064 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T08:55:15.105 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T08:55:15.105 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T08:55:15.170 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T08:55:15.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T08:55:15.261 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T08:55:15.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T08:55:15.382 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T08:55:15.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T08:55:15.446 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-09-17T08:55:15.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-09-17T08:55:15.519 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T08:55:15.534 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T08:55:16.047 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T08:55:16.055 INFO:teuthology.task.internal:Starting timer... 2024-09-17T08:55:16.055 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T08:55:16.069 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T08:55:16.077 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-17T08:55:16.078 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-09-17T08:55:16.139 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-09-17T08:55:16.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-09-17T08:55:16.204 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-09-17T08:55:16.206 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T08:55:16.206 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-09-17T08:55:16.244 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-09-17T08:55:16.244 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-09-17T08:55:16.264 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-09-17T08:55:16.264 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-09-17T08:55:16.265 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T08:55:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T08:55:16.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T08:55:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T08:55:16.339 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T08:55:16.339 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-09-17T08:55:16.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-09-17T08:55:16.411 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T08:55:16.420 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T08:55:16.432 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T08:55:16.433 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T08:59:05.046 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-09-17T08:59:05.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-17T08:59:05.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T08:59:05.127 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-09-17T08:59:05.213 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-17T08:59:05.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T08:59:05.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T08:59:05.292 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-09-17T08:59:05.372 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T08:59:05.372 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T08:59:05.385 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T08:59:05.385 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T08:59:05.386 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T08:59:05.389 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T08:59:05.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T08:59:05.419 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T08:59:05.434 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T08:59:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T08:59:05.455 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T08:59:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-09-17T08:59:05.473 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-09-17T08:59:05.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-09-17T08:59:05.484 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T08:59:05.490 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-09-17T08:59:05.499 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T08:59:05.505 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T08:59:05.519 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T08:59:05.557 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-09-17T08:59:05.561 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T08:59:05.561 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-09-17T08:59:05.561 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T08:59:05.561 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T08:59:05.561 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T08:59:05.561 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T08:59:05.575 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-09-17T08:59:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T08:59:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-09-17T08:59:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T08:59:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T08:59:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T08:59:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T08:59:05.580 INFO:teuthology.run_tasks:Running task pexec... 2024-09-17T08:59:05.590 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-17T08:59:05.590 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi079.front.sepia.ceph.com 2024-09-17T08:59:05.590 DEBUG:teuthology.orchestra.run.smithi079:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-17T08:59:05.591 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T08:59:05.591 DEBUG:teuthology.orchestra.run.smithi173:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-17T08:59:05.950 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout:Removing: 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout:Removing unused dependencies: 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout:Remove 2 Packages 2024-09-17T08:59:05.956 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:05.957 INFO:teuthology.orchestra.run.smithi079.stdout:Freed space: 5.5 M 2024-09-17T08:59:05.957 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T08:59:05.960 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T08:59:05.960 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T08:59:05.973 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout:Removing: 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout:Removing unused dependencies: 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T08:59:05.986 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:59:05.987 INFO:teuthology.orchestra.run.smithi173.stdout:Remove 2 Packages 2024-09-17T08:59:05.987 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:05.987 INFO:teuthology.orchestra.run.smithi173.stdout:Freed space: 5.5 M 2024-09-17T08:59:05.987 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T08:59:05.990 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T08:59:05.991 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T08:59:06.020 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T08:59:06.021 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T08:59:06.058 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T08:59:06.058 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T08:59:06.153 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T08:59:06.204 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T08:59:06.236 INFO:teuthology.orchestra.run.smithi079.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-17T08:59:06.284 INFO:teuthology.orchestra.run.smithi173.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-17T08:59:06.300 INFO:teuthology.orchestra.run.smithi079.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-17T08:59:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-17T08:59:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-17T08:59:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-17T08:59:06.828 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-17T08:59:06.828 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-17T08:59:06.965 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-17T08:59:06.966 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:06.966 INFO:teuthology.orchestra.run.smithi173.stdout:Removed: 2024-09-17T08:59:06.966 INFO:teuthology.orchestra.run.smithi173.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-17T08:59:06.966 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:06.966 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T08:59:07.102 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-17T08:59:07.102 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:07.102 INFO:teuthology.orchestra.run.smithi079.stdout:Removed: 2024-09-17T08:59:07.103 INFO:teuthology.orchestra.run.smithi079.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-17T08:59:07.103 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:07.103 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T08:59:07.851 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 08:58:15 AM UTC. 2024-09-17T08:59:07.939 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:44 ago on Tue 17 Sep 2024 08:58:23 AM UTC. 2024-09-17T08:59:08.003 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T08:59:08.003 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-17T08:59:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-17T08:59:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:08.005 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T08:59:08.005 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:59:08.005 INFO:teuthology.orchestra.run.smithi173.stdout:Install 7 Packages 2024-09-17T08:59:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:08.017 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 5.2 M 2024-09-17T08:59:08.018 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 19 M 2024-09-17T08:59:08.018 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T08:59:08.075 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T08:59:08.076 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:08.076 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T08:59:08.076 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:08.076 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T08:59:08.076 INFO:teuthology.orchestra.run.smithi079.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-17T08:59:08.076 INFO:teuthology.orchestra.run.smithi079.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-17T08:59:08.076 INFO:teuthology.orchestra.run.smithi079.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout:Install 7 Packages 2024-09-17T08:59:08.077 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:08.083 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 5.2 M 2024-09-17T08:59:08.083 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 19 M 2024-09-17T08:59:08.083 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T08:59:08.235 INFO:teuthology.orchestra.run.smithi173.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 376 kB/s | 44 kB 00:00 2024-09-17T08:59:08.261 INFO:teuthology.orchestra.run.smithi173.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 739 kB/s | 106 kB 00:00 2024-09-17T08:59:08.311 INFO:teuthology.orchestra.run.smithi173.stdout:(3/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.7 MB/s | 84 kB 00:00 2024-09-17T08:59:08.336 INFO:teuthology.orchestra.run.smithi173.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 713 kB/s | 72 kB 00:00 2024-09-17T08:59:08.341 INFO:teuthology.orchestra.run.smithi079.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 444 kB/s | 44 kB 00:00 2024-09-17T08:59:08.367 INFO:teuthology.orchestra.run.smithi079.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 845 kB/s | 106 kB 00:00 2024-09-17T08:59:08.378 INFO:teuthology.orchestra.run.smithi173.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.4 MB/s | 906 kB 00:00 2024-09-17T08:59:08.400 INFO:teuthology.orchestra.run.smithi079.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 1.2 MB/s | 72 kB 00:00 2024-09-17T08:59:08.426 INFO:teuthology.orchestra.run.smithi079.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-17T08:59:08.493 INFO:teuthology.orchestra.run.smithi079.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.5 MB/s | 906 kB 00:00 2024-09-17T08:59:08.518 INFO:teuthology.orchestra.run.smithi079.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 7.0 MB/s | 837 kB 00:00 2024-09-17T08:59:08.520 INFO:teuthology.orchestra.run.smithi173.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 3.9 MB/s | 837 kB 00:00 2024-09-17T08:59:09.127 INFO:teuthology.orchestra.run.smithi079.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 4.5 MB/s | 3.2 MB 00:00 2024-09-17T08:59:09.128 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T08:59:09.128 INFO:teuthology.orchestra.run.smithi079.stdout:Total 5.0 MB/s | 5.2 MB 00:01 2024-09-17T08:59:09.236 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T08:59:09.251 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T08:59:09.251 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T08:59:09.421 INFO:teuthology.orchestra.run.smithi173.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 2.9 MB/s | 3.2 MB 00:01 2024-09-17T08:59:09.421 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T08:59:09.421 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.7 MB/s | 5.2 MB 00:01 2024-09-17T08:59:09.530 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T08:59:09.541 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T08:59:09.542 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T08:59:09.544 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T08:59:09.544 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T08:59:09.862 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T08:59:09.862 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T08:59:10.013 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T08:59:10.113 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-17T08:59:10.148 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-17T08:59:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-17T08:59:10.322 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T08:59:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-17T08:59:10.414 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-17T08:59:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-17T08:59:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-17T08:59:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-17T08:59:10.695 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-17T08:59:10.725 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-17T08:59:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-17T08:59:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-17T08:59:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-17T08:59:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-17T08:59:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-17T08:59:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-17T08:59:11.317 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-17T08:59:11.382 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-17T08:59:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-17T08:59:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-17T08:59:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-17T08:59:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-17T08:59:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-17T08:59:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-17T08:59:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-17T08:59:12.353 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-17T08:59:12.353 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-17T08:59:12.353 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-17T08:59:12.353 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-17T08:59:12.354 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-17T08:59:12.354 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-17T08:59:12.354 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-17T08:59:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-17T08:59:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:12.360 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T08:59:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-17T08:59:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-17T08:59:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-17T08:59:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-17T08:59:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:12.360 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T08:59:12.746 DEBUG:teuthology.parallel:result is None 2024-09-17T08:59:12.764 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-17T08:59:12.764 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:12.764 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T08:59:12.764 INFO:teuthology.orchestra.run.smithi173.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-17T08:59:12.765 INFO:teuthology.orchestra.run.smithi173.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-17T08:59:12.765 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-17T08:59:12.765 INFO:teuthology.orchestra.run.smithi173.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-17T08:59:12.765 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:12.765 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T08:59:13.128 DEBUG:teuthology.parallel:result is None 2024-09-17T08:59:13.128 INFO:teuthology.run_tasks:Running task install... 2024-09-17T08:59:13.138 DEBUG:teuthology.task.install:project ceph 2024-09-17T08:59:13.138 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}} 2024-09-17T08:59:13.138 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'} 2024-09-17T08:59:13.138 INFO:teuthology.task.install:Using flavor: default 2024-09-17T08:59:13.144 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T08:59:13.144 INFO:teuthology.task.install:extra packages: [] 2024-09-17T08:59:13.144 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'tag': None, 'wait_for_package': False} 2024-09-17T08:59:13.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T08:59:13.147 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'tag': None, 'wait_for_package': False} 2024-09-17T08:59:13.147 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T08:59:13.296 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/ 2024-09-17T08:59:13.297 INFO:teuthology.task.install.rpm:Package version is 18.2.4-855.g19e2a2be 2024-09-17T08:59:13.300 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/ 2024-09-17T08:59:13.301 INFO:teuthology.task.install.rpm:Package version is 18.2.4-855.g19e2a2be 2024-09-17T08:59:13.408 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T08:59:13.408 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T08:59:13.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T08:59:13.417 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T08:59:13.417 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T08:59:13.417 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T08:59:13.434 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T08:59:13.434 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T08:59:13.445 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T08:59:13.445 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T08:59:13.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T08:59:13.513 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T08:59:13.574 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T08:59:13.590 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T08:59:13.652 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-09-17T08:59:13.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-09-17T08:59:13.655 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-09-17T08:59:13.657 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-09-17T08:59:13.880 INFO:teuthology.orchestra.run.smithi173.stdout:45 files removed 2024-09-17T08:59:13.888 INFO:teuthology.orchestra.run.smithi079.stdout:45 files removed 2024-09-17T08:59:13.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-09-17T08:59:13.914 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2024-09-17T08:59:14.453 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 312 kB/s | 77 kB 00:00 2024-09-17T08:59:14.460 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 290 kB/s | 77 kB 00:00 2024-09-17T08:59:14.729 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 49 kB/s | 11 kB 00:00 2024-09-17T08:59:14.744 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-17T08:59:14.989 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 8.4 kB/s | 1.9 kB 00:00 2024-09-17T08:59:15.028 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-17T08:59:15.687 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.3 MB 00:00 2024-09-17T08:59:17.022 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.3 MB 00:01 2024-09-17T08:59:20.081 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-09-17T08:59:24.482 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-17T08:59:26.066 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 19 kB 00:00 2024-09-17T08:59:27.046 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-17T08:59:32.478 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 20 MB 00:15 2024-09-17T08:59:32.493 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-09-17T08:59:34.745 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 7.7 M 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.3 M 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T08:59:34.747 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 5.1 M 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 18 M 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 25 k 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 712 k 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 477 k 2024-09-17T08:59:34.748 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.5 M 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 45 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 129 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 162 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 321 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 300 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 99 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T08:59:34.749 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T08:59:34.750 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:34.750 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T08:59:34.750 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:34.750 INFO:teuthology.orchestra.run.smithi079.stdout:Install 28 Packages 2024-09-17T08:59:34.750 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-09-17T08:59:34.750 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:34.750 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 53 M 2024-09-17T08:59:34.750 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T08:59:35.628 INFO:teuthology.orchestra.run.smithi079.stdout:(1/30): ceph-base-18.2.4-855.g19e2a2be.el9.x86_ 8.6 MB/s | 5.1 MB 00:00 2024-09-17T08:59:35.720 INFO:teuthology.orchestra.run.smithi079.stdout:(2/30): ceph-radosgw-18.2.4-855.g19e2a2be.el9.x 11 MB/s | 7.7 MB 00:00 2024-09-17T08:59:35.745 INFO:teuthology.orchestra.run.smithi079.stdout:(3/30): ceph-selinux-18.2.4-855.g19e2a2be.el9.x 215 kB/s | 25 kB 00:00 2024-09-17T08:59:35.779 INFO:teuthology.orchestra.run.smithi079.stdout:(4/30): libcephfs2-18.2.4-855.g19e2a2be.el9.x86 12 MB/s | 712 kB 00:00 2024-09-17T08:59:35.812 INFO:teuthology.orchestra.run.smithi079.stdout:(5/30): libradosstriper1-18.2.4-855.g19e2a2be.e 6.9 MB/s | 477 kB 00:00 2024-09-17T08:59:35.871 INFO:teuthology.orchestra.run.smithi079.stdout:(6/30): python3-ceph-argparse-18.2.4-855.g19e2a 773 kB/s | 45 kB 00:00 2024-09-17T08:59:35.946 INFO:teuthology.orchestra.run.smithi079.stdout:(7/30): python3-ceph-common-18.2.4-855.g19e2a2b 1.7 MB/s | 129 kB 00:00 2024-09-17T08:59:36.005 INFO:teuthology.orchestra.run.smithi079.stdout:(8/30): librgw2-18.2.4-855.g19e2a2be.el9.x86_64 20 MB/s | 4.5 MB 00:00 2024-09-17T08:59:36.030 INFO:teuthology.orchestra.run.smithi079.stdout:(9/30): python3-cephfs-18.2.4-855.g19e2a2be.el9 1.9 MB/s | 162 kB 00:00 2024-09-17T08:59:36.064 INFO:teuthology.orchestra.run.smithi079.stdout:(10/30): python3-rados-18.2.4-855.g19e2a2be.el9 5.3 MB/s | 321 kB 00:00 2024-09-17T08:59:36.239 INFO:teuthology.orchestra.run.smithi079.stdout:(11/30): ceph-common-18.2.4-855.g19e2a2be.el9.x 15 MB/s | 18 MB 00:01 2024-09-17T08:59:36.273 INFO:teuthology.orchestra.run.smithi079.stdout:(12/30): python3-rbd-18.2.4-855.g19e2a2be.el9.x 1.2 MB/s | 300 kB 00:00 2024-09-17T08:59:36.298 INFO:teuthology.orchestra.run.smithi079.stdout:(13/30): python3-rgw-18.2.4-855.g19e2a2be.el9.x 425 kB/s | 99 kB 00:00 2024-09-17T08:59:36.415 INFO:teuthology.orchestra.run.smithi079.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 388 kB/s | 45 kB 00:00 2024-09-17T08:59:36.440 INFO:teuthology.orchestra.run.smithi079.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 533 kB/s | 107 kB 00:00 2024-09-17T08:59:36.466 INFO:teuthology.orchestra.run.smithi079.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 832 kB/s | 160 kB 00:00 2024-09-17T08:59:36.516 INFO:teuthology.orchestra.run.smithi079.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-17T08:59:36.542 INFO:teuthology.orchestra.run.smithi079.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-17T08:59:36.575 INFO:teuthology.orchestra.run.smithi079.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-17T08:59:36.634 INFO:teuthology.orchestra.run.smithi079.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 945 kB/s | 111 kB 00:00 2024-09-17T08:59:36.651 INFO:teuthology.orchestra.run.smithi079.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.7 MB/s | 26 kB 00:00 2024-09-17T08:59:36.668 INFO:teuthology.orchestra.run.smithi079.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 2.8 MB/s | 49 kB 00:00 2024-09-17T08:59:36.687 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-17T08:59:36.693 INFO:teuthology.orchestra.run.smithi079.stdout:(23/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-17T08:59:36.719 INFO:teuthology.orchestra.run.smithi079.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T08:59:36.770 INFO:teuthology.orchestra.run.smithi079.stdout:(25/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-09-17T08:59:36.795 INFO:teuthology.orchestra.run.smithi079.stdout:(26/30): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-17T08:59:36.878 INFO:teuthology.orchestra.run.smithi079.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-17T08:59:36.945 INFO:teuthology.orchestra.run.smithi079.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-17T08:59:37.054 INFO:teuthology.orchestra.run.smithi079.stdout:(29/30): librados2-18.2.4-855.g19e2a2be.el9.x86 13 MB/s | 3.3 MB 00:00 2024-09-17T08:59:37.112 INFO:teuthology.orchestra.run.smithi079.stdout:(30/30): librbd1-18.2.4-855.g19e2a2be.el9.x86_6 13 MB/s | 3.0 MB 00:00 2024-09-17T08:59:37.117 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T08:59:37.117 INFO:teuthology.orchestra.run.smithi079.stdout:Total 22 MB/s | 53 MB 00:02 2024-09-17T08:59:37.189 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T08:59:37.223 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T08:59:37.223 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T08:59:37.680 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T08:59:37.681 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T08:59:38.387 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 19 kB 00:00 2024-09-17T08:59:39.022 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T08:59:39.115 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-17T08:59:39.151 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-17T08:59:39.202 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-17T08:59:39.258 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-17T08:59:39.523 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-17T08:59:39.634 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T08:59:39.726 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T08:59:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-17T08:59:39.861 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T08:59:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T08:59:39.952 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-17T08:59:40.004 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 9/32 2024-09-17T08:59:40.070 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-17T08:59:40.097 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T08:59:40.146 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T08:59:40.253 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-17T08:59:40.288 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-17T08:59:40.329 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-17T08:59:40.370 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-17T08:59:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-17T08:59:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-17T08:59:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-17T08:59:40.778 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-17T08:59:40.820 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-17T08:59:40.903 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-17T08:59:40.934 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 21/32 2024-09-17T08:59:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-17T08:59:41.377 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-17T08:59:41.407 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-17T08:59:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-17T08:59:41.538 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 25/32 2024-09-17T08:59:41.568 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 26/32 2024-09-17T08:59:43.690 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T08:59:43.732 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T08:59:44.275 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T08:59:44.311 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-17T08:59:44.423 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-17T08:59:44.423 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T08:59:44.423 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T08:59:44.423 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:44.454 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T08:59:44.614 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T08:59:46.838 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T08:59:46.840 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:59:46.840 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T08:59:46.840 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:59:46.840 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T08:59:46.840 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 7.7 M 2024-09-17T08:59:46.840 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.3 M 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 5.1 M 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 18 M 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 25 k 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T08:59:46.841 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 712 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 477 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.5 M 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 45 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 129 k 2024-09-17T08:59:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 162 k 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 321 k 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 300 k 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 99 k 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T08:59:46.843 INFO:teuthology.orchestra.run.smithi173.stdout:Install 28 Packages 2024-09-17T08:59:46.844 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-09-17T08:59:46.844 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:46.844 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 53 M 2024-09-17T08:59:46.844 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T08:59:47.749 INFO:teuthology.orchestra.run.smithi173.stdout:(1/30): ceph-base-18.2.4-855.g19e2a2be.el9.x86_ 6.5 MB/s | 5.1 MB 00:00 2024-09-17T08:59:47.833 INFO:teuthology.orchestra.run.smithi173.stdout:(2/30): ceph-selinux-18.2.4-855.g19e2a2be.el9.x 301 kB/s | 25 kB 00:00 2024-09-17T08:59:47.949 INFO:teuthology.orchestra.run.smithi173.stdout:(3/30): ceph-radosgw-18.2.4-855.g19e2a2be.el9.x 7.9 MB/s | 7.7 MB 00:00 2024-09-17T08:59:48.017 INFO:teuthology.orchestra.run.smithi173.stdout:(4/30): libcephfs2-18.2.4-855.g19e2a2be.el9.x86 3.8 MB/s | 712 kB 00:00 2024-09-17T08:59:48.042 INFO:teuthology.orchestra.run.smithi173.stdout:(5/30): libradosstriper1-18.2.4-855.g19e2a2be.e 5.1 MB/s | 477 kB 00:00 2024-09-17T08:59:48.084 INFO:teuthology.orchestra.run.smithi173.stdout:(6/30): python3-ceph-argparse-18.2.4-855.g19e2a 1.1 MB/s | 45 kB 00:00 2024-09-17T08:59:48.142 INFO:teuthology.orchestra.run.smithi173.stdout:(7/30): python3-ceph-common-18.2.4-855.g19e2a2b 2.2 MB/s | 129 kB 00:00 2024-09-17T08:59:48.209 INFO:teuthology.orchestra.run.smithi173.stdout:(8/30): python3-cephfs-18.2.4-855.g19e2a2be.el9 2.4 MB/s | 162 kB 00:00 2024-09-17T08:59:48.310 INFO:teuthology.orchestra.run.smithi173.stdout:(9/30): librgw2-18.2.4-855.g19e2a2be.el9.x86_64 15 MB/s | 4.5 MB 00:00 2024-09-17T08:59:48.335 INFO:teuthology.orchestra.run.smithi173.stdout:(10/30): python3-rados-18.2.4-855.g19e2a2be.el9 2.5 MB/s | 321 kB 00:00 2024-09-17T08:59:48.360 INFO:teuthology.orchestra.run.smithi173.stdout:(11/30): python3-rbd-18.2.4-855.g19e2a2be.el9.x 5.9 MB/s | 300 kB 00:00 2024-09-17T08:59:48.386 INFO:teuthology.orchestra.run.smithi173.stdout:(12/30): python3-rgw-18.2.4-855.g19e2a2be.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-17T08:59:48.603 INFO:teuthology.orchestra.run.smithi173.stdout:(13/30): ceph-common-18.2.4-855.g19e2a2be.el9.x 11 MB/s | 18 MB 00:01 2024-09-17T08:59:48.712 INFO:teuthology.orchestra.run.smithi173.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 419 kB/s | 45 kB 00:00 2024-09-17T08:59:48.737 INFO:teuthology.orchestra.run.smithi173.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 284 kB/s | 107 kB 00:00 2024-09-17T08:59:48.762 INFO:teuthology.orchestra.run.smithi173.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 425 kB/s | 160 kB 00:00 2024-09-17T08:59:48.812 INFO:teuthology.orchestra.run.smithi173.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 849 kB/s | 42 kB 00:00 2024-09-17T08:59:48.846 INFO:teuthology.orchestra.run.smithi173.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-17T08:59:48.880 INFO:teuthology.orchestra.run.smithi173.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-09-17T08:59:48.930 INFO:teuthology.orchestra.run.smithi173.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 943 kB/s | 111 kB 00:00 2024-09-17T08:59:48.955 INFO:teuthology.orchestra.run.smithi173.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-17T08:59:48.989 INFO:teuthology.orchestra.run.smithi173.stdout:(22/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-17T08:59:49.014 INFO:teuthology.orchestra.run.smithi173.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-09-17T08:59:49.039 INFO:teuthology.orchestra.run.smithi173.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T08:59:49.073 INFO:teuthology.orchestra.run.smithi173.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T08:59:49.115 INFO:teuthology.orchestra.run.smithi173.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-17T08:59:49.190 INFO:teuthology.orchestra.run.smithi173.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-17T08:59:49.274 INFO:teuthology.orchestra.run.smithi173.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-17T08:59:49.424 INFO:teuthology.orchestra.run.smithi173.stdout:(29/30): librados2-18.2.4-855.g19e2a2be.el9.x86 11 MB/s | 3.3 MB 00:00 2024-09-17T08:59:49.549 INFO:teuthology.orchestra.run.smithi173.stdout:(30/30): librbd1-18.2.4-855.g19e2a2be.el9.x86_6 8.4 MB/s | 3.0 MB 00:00 2024-09-17T08:59:49.554 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T08:59:49.555 INFO:teuthology.orchestra.run.smithi173.stdout:Total 20 MB/s | 53 MB 00:02 2024-09-17T08:59:49.622 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T08:59:49.657 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T08:59:49.657 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T08:59:50.098 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T08:59:50.099 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T08:59:51.438 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T08:59:51.517 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-17T08:59:51.574 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-17T08:59:51.637 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-17T08:59:51.958 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-17T08:59:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T08:59:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T08:59:52.246 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-17T08:59:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T08:59:52.343 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T08:59:52.371 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-17T08:59:52.415 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 9/32 2024-09-17T08:59:52.489 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-17T08:59:52.515 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T08:59:52.581 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T08:59:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-17T08:59:52.756 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-17T08:59:52.797 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-17T08:59:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-17T08:59:52.865 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-17T08:59:52.901 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-17T08:59:52.934 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-17T08:59:53.240 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-17T08:59:53.280 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-17T08:59:53.354 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-17T08:59:53.384 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 21/32 2024-09-17T08:59:53.421 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-17T08:59:53.834 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-17T08:59:53.865 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-17T08:59:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-17T08:59:54.013 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 25/32 2024-09-17T08:59:54.043 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 26/32 2024-09-17T08:59:54.847 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T08:59:54.847 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-09-17T08:59:54.847 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-09-17T08:59:54.847 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-09-17T08:59:54.847 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-09-17T08:59:54.847 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-09-17T08:59:54.847 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-09-17T08:59:54.847 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-09-17T08:59:54.848 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:55.625 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-17T08:59:55.650 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-17T08:59:55.650 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T08:59:55.650 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T08:59:55.650 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T08:59:55.650 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T08:59:55.650 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:55.652 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-17T08:59:55.736 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-17T08:59:55.737 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T08:59:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T08:59:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T08:59:56.748 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T08:59:56.793 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-17T08:59:56.891 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-17T08:59:56.892 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T08:59:56.892 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T08:59:56.892 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T08:59:56.924 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T08:59:56.924 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 1/32 2024-09-17T08:59:56.924 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 3/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 4/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 8/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 9/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T08:59:56.925 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 12/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 13/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-17T08:59:56.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-17T08:59:56.927 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-17T08:59:56.927 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-17T08:59:56.927 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-17T08:59:56.927 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-17T08:59:56.927 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-17T08:59:56.927 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T08:59:56.927 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-17T08:59:56.927 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 31/32 2024-09-17T08:59:56.937 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T08:59:57.985 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T08:59:57.985 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:57.985 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-09-17T08:59:57.985 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.985 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.985 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T08:59:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T08:59:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T08:59:57.988 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:57.988 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T08:59:58.424 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2024-09-17T08:59:59.001 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:26 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T08:59:59.133 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T08:59:59.133 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:59.133 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T08:59:59.133 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:59.133 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 40 M 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 166 k 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-09-17T08:59:59.134 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T08:59:59.135 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 40 M 2024-09-17T08:59:59.135 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 164 M 2024-09-17T08:59:59.135 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T08:59:59.604 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): libcephsqlite-18.2.4-855.g19e2a2be.el9.x 775 kB/s | 166 kB 00:00 2024-09-17T08:59:59.638 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-17T08:59:59.755 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 424 kB/s | 64 kB 00:00 2024-09-17T09:00:00.705 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): ceph-test-18.2.4-855.g19e2a2be.el9.x86_6 30 MB/s | 40 MB 00:01 2024-09-17T09:00:00.707 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:00.707 INFO:teuthology.orchestra.run.smithi079.stdout:Total 26 MB/s | 40 MB 00:01 2024-09-17T09:00:00.737 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:00:00.757 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:00:00.757 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:00:01.083 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:00:01.084 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:00:01.573 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:00:01.680 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T09:00:01.730 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T09:00:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-17T09:00:05.445 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-17T09:00:05.531 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-17T09:00:07.062 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T09:00:07.062 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-09-17T09:00:07.062 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-09-17T09:00:07.063 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-09-17T09:00:07.063 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-09-17T09:00:07.063 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-09-17T09:00:07.063 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-09-17T09:00:07.063 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-09-17T09:00:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:07.545 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-17T09:00:07.546 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 1/4 2024-09-17T09:00:07.546 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2/4 2024-09-17T09:00:07.546 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T09:00:07.860 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-17T09:00:07.883 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T09:00:07.883 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:07.883 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:00:07.884 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:07.884 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:07.884 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T09:00:07.884 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T09:00:07.884 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:07.884 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:00:07.885 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-17T09:00:07.885 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:07.885 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T09:00:07.885 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T09:00:07.886 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T09:00:07.886 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:07.887 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-17T09:00:07.971 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-17T09:00:07.971 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T09:00:08.224 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2024-09-17T09:00:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:36 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:00:08.947 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:00:08.953 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:08.953 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 6.5 k 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 2.1 M 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 1.5 M 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 246 k 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.7 M 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 17 M 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T09:00:08.954 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T09:00:08.955 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:08.956 INFO:teuthology.orchestra.run.smithi079.stdout:Install 33 Packages 2024-09-17T09:00:08.957 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:08.957 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 29 M 2024-09-17T09:00:08.957 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 104 M 2024-09-17T09:00:08.957 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:00:09.128 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T09:00:09.128 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 1/32 2024-09-17T09:00:09.128 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2/32 2024-09-17T09:00:09.128 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 3/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 4/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 8/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 9/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 12/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 13/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-17T09:00:09.129 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-17T09:00:09.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 31/32 2024-09-17T09:00:09.486 INFO:teuthology.orchestra.run.smithi079.stdout:(1/33): ceph-18.2.4-855.g19e2a2be.el9.x86_64.rp 55 kB/s | 6.5 kB 00:00 2024-09-17T09:00:09.694 INFO:teuthology.orchestra.run.smithi079.stdout:(2/33): ceph-mgr-18.2.4-855.g19e2a2be.el9.x86_6 4.5 MB/s | 1.5 MB 00:00 2024-09-17T09:00:09.745 INFO:teuthology.orchestra.run.smithi079.stdout:(3/33): ceph-mds-18.2.4-855.g19e2a2be.el9.x86_6 5.6 MB/s | 2.1 MB 00:00 2024-09-17T09:00:09.820 INFO:teuthology.orchestra.run.smithi079.stdout:(4/33): ceph-mgr-modules-core-18.2.4-855.g19e2a 3.2 MB/s | 246 kB 00:00 2024-09-17T09:00:10.012 INFO:teuthology.orchestra.run.smithi079.stdout:(5/33): ceph-mon-18.2.4-855.g19e2a2be.el9.x86_6 8.9 MB/s | 4.7 MB 00:00 2024-09-17T09:00:10.042 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T09:00:10.042 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T09:00:10.043 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.045 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T09:00:10.045 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.045 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.045 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:10.045 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T09:00:10.045 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T09:00:10.045 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:10.045 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:00:10.296 INFO:teuthology.orchestra.run.smithi079.stdout:(6/33): ceph-osd-18.2.4-855.g19e2a2be.el9.x86_6 27 MB/s | 17 MB 00:00 2024-09-17T09:00:10.321 INFO:teuthology.orchestra.run.smithi079.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 506 kB/s | 253 kB 00:00 2024-09-17T09:00:10.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-09-17T09:00:10.455 INFO:teuthology.orchestra.run.smithi079.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.0 MB/s | 135 kB 00:00 2024-09-17T09:00:10.580 INFO:teuthology.orchestra.run.smithi079.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 998 kB/s | 125 kB 00:00 2024-09-17T09:00:10.606 INFO:teuthology.orchestra.run.smithi079.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 344 kB/s | 106 kB 00:00 2024-09-17T09:00:10.748 INFO:teuthology.orchestra.run.smithi079.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-09-17T09:00:10.995 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:26 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:00:10.998 INFO:teuthology.orchestra.run.smithi079.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:00 2024-09-17T09:00:11.126 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 40 M 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 166 k 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T09:00:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:11.128 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:00:11.128 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:11.128 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-09-17T09:00:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:11.128 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 40 M 2024-09-17T09:00:11.128 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 164 M 2024-09-17T09:00:11.128 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:00:11.281 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-17T09:00:11.321 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.6 MB/s | 64 kB 00:00 2024-09-17T09:00:11.348 INFO:teuthology.orchestra.run.smithi079.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 232 kB/s | 172 kB 00:00 2024-09-17T09:00:11.380 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): libcephsqlite-18.2.4-855.g19e2a2be.el9.x 745 kB/s | 166 kB 00:00 2024-09-17T09:00:11.381 INFO:teuthology.orchestra.run.smithi079.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 55 kB/s | 35 kB 00:00 2024-09-17T09:00:11.424 INFO:teuthology.orchestra.run.smithi079.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-09-17T09:00:11.449 INFO:teuthology.orchestra.run.smithi079.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-09-17T09:00:11.482 INFO:teuthology.orchestra.run.smithi079.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 10 MB/s | 358 kB 00:00 2024-09-17T09:00:11.508 INFO:teuthology.orchestra.run.smithi079.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 426 kB/s | 11 kB 00:00 2024-09-17T09:00:11.533 INFO:teuthology.orchestra.run.smithi079.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 712 kB/s | 18 kB 00:00 2024-09-17T09:00:11.558 INFO:teuthology.orchestra.run.smithi079.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 926 kB/s | 23 kB 00:00 2024-09-17T09:00:11.584 INFO:teuthology.orchestra.run.smithi079.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 776 kB/s | 19 kB 00:00 2024-09-17T09:00:11.642 INFO:teuthology.orchestra.run.smithi079.stdout:(22/33): python3-pytz-2021.1-5.el9.noarch.rpm 80 kB/s | 51 kB 00:00 2024-09-17T09:00:11.668 INFO:teuthology.orchestra.run.smithi079.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 258 kB/s | 20 kB 00:00 2024-09-17T09:00:11.701 INFO:teuthology.orchestra.run.smithi079.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 3.0 MB/s | 79 kB 00:00 2024-09-17T09:00:11.735 INFO:teuthology.orchestra.run.smithi079.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 8.5 MB/s | 272 kB 00:00 2024-09-17T09:00:11.769 INFO:teuthology.orchestra.run.smithi079.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 367 kB/s | 46 kB 00:00 2024-09-17T09:00:11.794 INFO:teuthology.orchestra.run.smithi079.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 279 kB/s | 16 kB 00:00 2024-09-17T09:00:11.819 INFO:teuthology.orchestra.run.smithi079.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-17T09:00:11.845 INFO:teuthology.orchestra.run.smithi079.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-17T09:00:11.870 INFO:teuthology.orchestra.run.smithi079.stdout:(30/33): python3-webob-1.8.8-2.el9.noarch.rpm 4.4 MB/s | 230 kB 00:00 2024-09-17T09:00:11.887 INFO:teuthology.orchestra.run.smithi079.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-17T09:00:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.2 MB/s | 427 kB 00:00 2024-09-17T09:00:11.963 INFO:teuthology.orchestra.run.smithi079.stdout:(33/33): python3-toml-0.10.2-6.el9.noarch.rpm 68 kB/s | 42 kB 00:00 2024-09-17T09:00:11.968 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:11.968 INFO:teuthology.orchestra.run.smithi079.stdout:Total 9.7 MB/s | 29 MB 00:03 2024-09-17T09:00:12.090 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:00:12.129 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:00:12.129 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:00:12.472 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): ceph-test-18.2.4-855.g19e2a2be.el9.x86_6 30 MB/s | 40 MB 00:01 2024-09-17T09:00:12.474 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:12.474 INFO:teuthology.orchestra.run.smithi173.stdout:Total 30 MB/s | 40 MB 00:01 2024-09-17T09:00:12.500 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:00:12.519 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:00:12.519 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:00:12.528 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:00:12.529 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:00:12.851 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:00:12.851 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:00:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:00:13.429 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T09:00:13.471 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T09:00:13.498 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-17T09:00:13.776 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:00:13.832 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-17T09:00:13.865 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-17T09:00:13.897 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-17T09:00:13.998 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-17T09:00:14.076 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-17T09:00:14.118 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-17T09:00:14.152 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-17T09:00:14.191 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-17T09:00:14.225 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-17T09:00:14.259 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-17T09:00:14.299 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-17T09:00:14.335 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-17T09:00:14.369 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-17T09:00:14.407 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-17T09:00:14.489 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-17T09:00:14.627 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-17T09:00:14.707 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-17T09:00:14.769 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-17T09:00:14.817 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-17T09:00:14.865 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-17T09:00:14.921 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-17T09:00:15.107 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-17T09:00:15.171 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-17T09:00:15.241 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-17T09:00:15.338 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-17T09:00:15.388 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-17T09:00:15.504 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-17T09:00:15.662 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 28/33 2024-09-17T09:00:15.690 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-17T09:00:15.715 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-17T09:00:15.715 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:15.715 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T09:00:15.716 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T09:00:15.716 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T09:00:15.716 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-17T09:00:17.213 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-17T09:00:17.214 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:17.214 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T09:00:17.214 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T09:00:17.214 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T09:00:17.214 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-17T09:00:17.314 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-17T09:00:17.685 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-17T09:00:17.709 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-17T09:00:17.709 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:17.709 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T09:00:17.710 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T09:00:17.710 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T09:00:17.710 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:17.911 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-17T09:00:17.934 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-17T09:00:17.934 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:17.934 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T09:00:17.934 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T09:00:17.935 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T09:00:17.935 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:18.000 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-17T09:00:19.491 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-17T09:00:19.491 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 1/4 2024-09-17T09:00:19.491 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2/4 2024-09-17T09:00:19.491 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T09:00:19.708 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-17T09:00:19.708 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 1/33 2024-09-17T09:00:19.708 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2/33 2024-09-17T09:00:19.708 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 3/33 2024-09-17T09:00:19.708 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 4/33 2024-09-17T09:00:19.708 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 5/33 2024-09-17T09:00:19.708 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 6/33 2024-09-17T09:00:19.708 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-17T09:00:19.708 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-17T09:00:19.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-17T09:00:19.710 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-17T09:00:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T09:00:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:19.839 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:00:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T09:00:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T09:00:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:19.839 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:00:20.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-09-17T09:00:20.647 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-17T09:00:20.647 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:20.647 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:00:20.647 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:20.647 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:20.647 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:20.647 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:20.647 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T09:00:20.648 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T09:00:20.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T09:00:20.650 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T09:00:20.650 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T09:00:20.650 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T09:00:20.650 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T09:00:20.650 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T09:00:20.650 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:20.650 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:00:20.792 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:36 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:00:20.927 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:00:20.929 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T09:00:20.929 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:20.929 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:00:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 6.5 k 2024-09-17T09:00:20.929 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T09:00:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 2.1 M 2024-09-17T09:00:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 1.5 M 2024-09-17T09:00:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 246 k 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.7 M 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 17 M 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T09:00:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T09:00:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T09:00:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T09:00:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T09:00:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T09:00:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T09:00:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:20.932 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:00:20.932 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:20.932 INFO:teuthology.orchestra.run.smithi173.stdout:Install 33 Packages 2024-09-17T09:00:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:20.933 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 29 M 2024-09-17T09:00:20.933 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 104 M 2024-09-17T09:00:20.933 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:00:21.090 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2024-09-17T09:00:21.672 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:49 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:00:21.716 INFO:teuthology.orchestra.run.smithi173.stdout:(1/33): ceph-18.2.4-855.g19e2a2be.el9.x86_64.rp 54 kB/s | 6.5 kB 00:00 2024-09-17T09:00:21.760 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:00:21.801 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:00:21.803 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-17T09:00:21.803 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:00:21.877 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2024-09-17T09:00:21.908 INFO:teuthology.orchestra.run.smithi173.stdout:(2/33): ceph-mds-18.2.4-855.g19e2a2be.el9.x86_6 6.7 MB/s | 2.1 MB 00:00 2024-09-17T09:00:21.950 INFO:teuthology.orchestra.run.smithi173.stdout:(3/33): ceph-mgr-18.2.4-855.g19e2a2be.el9.x86_6 4.1 MB/s | 1.5 MB 00:00 2024-09-17T09:00:22.034 INFO:teuthology.orchestra.run.smithi173.stdout:(4/33): ceph-mgr-modules-core-18.2.4-855.g19e2a 2.9 MB/s | 246 kB 00:00 2024-09-17T09:00:22.109 INFO:teuthology.orchestra.run.smithi173.stdout:(5/33): ceph-mon-18.2.4-855.g19e2a2be.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-17T09:00:22.453 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:50 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:00:22.468 INFO:teuthology.orchestra.run.smithi173.stdout:(6/33): ceph-osd-18.2.4-855.g19e2a2be.el9.x86_6 30 MB/s | 17 MB 00:00 2024-09-17T09:00:22.584 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 226 k 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 226 k 2024-09-17T09:00:22.585 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 217 k 2024-09-17T09:00:22.586 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:00:22.801 INFO:teuthology.orchestra.run.smithi173.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 330 kB/s | 253 kB 00:00 2024-09-17T09:00:22.828 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-18.2.4-855.g19e2a2be.el9.noarch.rpm 932 kB/s | 226 kB 00:00 2024-09-17T09:00:22.829 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:22.829 INFO:teuthology.orchestra.run.smithi079.stdout:Total 926 kB/s | 226 kB 00:00 2024-09-17T09:00:22.829 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:00:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:00:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:00:22.837 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:00:22.837 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:00:22.910 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:00:22.918 INFO:teuthology.orchestra.run.smithi173.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-17T09:00:22.968 INFO:teuthology.orchestra.run.smithi173.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 213 kB/s | 106 kB 00:00 2024-09-17T09:00:23.035 INFO:teuthology.orchestra.run.smithi173.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-17T09:00:23.094 INFO:teuthology.orchestra.run.smithi173.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:00 2024-09-17T09:00:23.152 INFO:teuthology.orchestra.run.smithi173.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-17T09:00:23.202 INFO:teuthology.orchestra.run.smithi173.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-17T09:00:23.261 INFO:teuthology.orchestra.run.smithi173.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-09-17T09:00:23.303 INFO:teuthology.orchestra.run.smithi173.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-09-17T09:00:23.328 INFO:teuthology.orchestra.run.smithi173.stdout:(16/33): python3-urllib3-1.26.5-6.el9.noarch.rp 597 kB/s | 215 kB 00:00 2024-09-17T09:00:23.341 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T09:00:23.353 INFO:teuthology.orchestra.run.smithi173.stdout:(17/33): python3-toml-0.10.2-6.el9.noarch.rpm 276 kB/s | 42 kB 00:00 2024-09-17T09:00:23.395 INFO:teuthology.orchestra.run.smithi173.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-09-17T09:00:23.421 INFO:teuthology.orchestra.run.smithi173.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 181 kB/s | 11 kB 00:00 2024-09-17T09:00:23.433 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T09:00:23.446 INFO:teuthology.orchestra.run.smithi173.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 366 kB/s | 18 kB 00:00 2024-09-17T09:00:23.472 INFO:teuthology.orchestra.run.smithi173.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-09-17T09:00:23.496 INFO:teuthology.orchestra.run.smithi173.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-17T09:00:23.522 INFO:teuthology.orchestra.run.smithi173.stdout:(23/33): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-09-17T09:00:23.547 INFO:teuthology.orchestra.run.smithi173.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-17T09:00:23.572 INFO:teuthology.orchestra.run.smithi173.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-09-17T09:00:23.606 INFO:teuthology.orchestra.run.smithi173.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 942 kB/s | 79 kB 00:00 2024-09-17T09:00:23.631 INFO:teuthology.orchestra.run.smithi173.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-09-17T09:00:23.656 INFO:teuthology.orchestra.run.smithi173.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-17T09:00:23.682 INFO:teuthology.orchestra.run.smithi173.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-17T09:00:23.707 INFO:teuthology.orchestra.run.smithi173.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-17T09:00:23.732 INFO:teuthology.orchestra.run.smithi173.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-17T09:00:23.757 INFO:teuthology.orchestra.run.smithi173.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-17T09:00:23.782 INFO:teuthology.orchestra.run.smithi173.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-17T09:00:23.783 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T09:00:23.787 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:23.787 INFO:teuthology.orchestra.run.smithi173.stdout:Total 10 MB/s | 29 MB 00:02 2024-09-17T09:00:23.910 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:00:23.949 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:00:23.949 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:00:23.992 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T09:00:23.993 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:23.993 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:00:23.993 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:23.993 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:23.993 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:00:24.182 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T09:00:24.353 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:00:24.354 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:00:24.768 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:00:24.903 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:00:24.903 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:24.903 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-09-17T09:00:24.903 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 143 k 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 143 k 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 395 k 2024-09-17T09:00:24.904 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:00:25.094 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-18.2.4-855.g19e2a2b 752 kB/s | 143 kB 00:00 2024-09-17T09:00:25.095 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:25.095 INFO:teuthology.orchestra.run.smithi079.stdout:Total 745 kB/s | 143 kB 00:00 2024-09-17T09:00:25.095 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:00:25.103 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:00:25.103 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:00:25.160 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:00:25.160 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:00:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:00:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T09:00:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T09:00:25.413 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:25.413 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T09:00:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:25.578 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:00:25.637 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-17T09:00:25.677 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-17T09:00:25.710 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-17T09:00:25.803 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-17T09:00:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-17T09:00:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-17T09:00:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-17T09:00:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-17T09:00:26.029 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-17T09:00:26.062 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-17T09:00:26.102 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-17T09:00:26.137 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-17T09:00:26.172 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-17T09:00:26.202 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-17T09:00:26.275 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-17T09:00:26.280 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T09:00:26.281 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:26.281 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:00:26.281 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:26.281 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:26.281 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:00:26.431 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-17T09:00:26.510 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2024-09-17T09:00:26.511 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-17T09:00:26.563 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-17T09:00:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-17T09:00:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-17T09:00:26.730 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-17T09:00:26.918 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-17T09:00:26.973 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-17T09:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-17T09:00:27.108 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:00:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-17T09:00:27.188 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-17T09:00:27.196 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:00:27.238 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:00:27.239 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-17T09:00:27.240 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:00:27.305 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-17T09:00:27.315 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T09:00:27.467 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 28/33 2024-09-17T09:00:27.498 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-17T09:00:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-17T09:00:27.523 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:27.524 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T09:00:27.524 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T09:00:27.524 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T09:00:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:27.897 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:00:28.029 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:00:28.030 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T09:00:28.030 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:28.030 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:00:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 3.5 M 2024-09-17T09:00:28.030 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T09:00:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 24 k 2024-09-17T09:00:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 16 k 2024-09-17T09:00:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T09:00:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T09:00:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:28.031 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:00:28.031 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:00:28.031 INFO:teuthology.orchestra.run.smithi079.stdout:Install 5 Packages 2024-09-17T09:00:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:28.032 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.8 M 2024-09-17T09:00:28.032 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 87 M 2024-09-17T09:00:28.032 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:00:28.250 INFO:teuthology.orchestra.run.smithi079.stdout:(1/5): ceph-grafana-dashboards-18.2.4-855.g19e2 153 kB/s | 24 kB 00:00 2024-09-17T09:00:28.276 INFO:teuthology.orchestra.run.smithi079.stdout:(2/5): ceph-prometheus-alerts-18.2.4-855.g19e2a 89 kB/s | 16 kB 00:00 2024-09-17T09:00:28.326 INFO:teuthology.orchestra.run.smithi079.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 408 kB/s | 31 kB 00:00 2024-09-17T09:00:28.351 INFO:teuthology.orchestra.run.smithi079.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-09-17T09:00:28.485 INFO:teuthology.orchestra.run.smithi079.stdout:(5/5): ceph-mgr-dashboard-18.2.4-855.g19e2a2be. 9.0 MB/s | 3.5 MB 00:00 2024-09-17T09:00:28.488 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:28.489 INFO:teuthology.orchestra.run.smithi079.stdout:Total 8.3 MB/s | 3.8 MB 00:00 2024-09-17T09:00:28.489 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:00:28.496 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:00:28.496 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:00:28.579 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:00:28.579 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:00:28.833 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:00:28.925 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-17T09:00:28.958 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-17T09:00:28.965 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-17T09:00:28.985 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-17T09:00:28.985 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:28.985 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T09:00:28.986 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T09:00:28.986 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T09:00:28.986 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:28.994 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-17T09:00:29.450 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-17T09:00:29.489 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-17T09:00:29.489 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:29.490 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T09:00:29.490 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T09:00:29.490 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T09:00:29.490 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:29.701 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-17T09:00:29.726 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-17T09:00:29.726 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:29.726 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T09:00:29.726 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T09:00:29.726 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T09:00:29.726 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:29.808 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-17T09:00:30.036 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 4/5 2024-09-17T09:00:30.081 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-17T09:00:31.197 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-17T09:00:31.197 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 1/5 2024-09-17T09:00:31.197 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2/5 2024-09-17T09:00:31.197 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-17T09:00:31.197 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-17T09:00:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-17T09:00:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:31.489 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:00:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T09:00:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T09:00:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:31.489 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 1/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 3/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 4/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 5/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 6/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-17T09:00:31.587 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-17T09:00:31.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-17T09:00:31.589 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-17T09:00:31.589 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-17T09:00:31.589 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-17T09:00:31.589 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-17T09:00:31.590 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-17T09:00:31.590 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-17T09:00:31.590 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-17T09:00:31.590 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-17T09:00:31.720 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T09:00:32.290 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:00:32.422 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 7.4 M 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T09:00:32.423 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-09-17T09:00:32.424 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:00:32.425 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 38 M 2024-09-17T09:00:32.425 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 212 M 2024-09-17T09:00:32.425 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:00:32.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-17T09:00:32.478 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:32.478 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T09:00:32.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T09:00:32.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:32.481 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:00:32.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-09-17T09:00:33.169 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.4-85 16 MB/s | 7.4 MB 00:00 2024-09-17T09:00:33.460 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 112 kB/s | 33 kB 00:00 2024-09-17T09:00:33.494 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:49 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:00:33.527 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 233 kB/s | 191 kB 00:00 2024-09-17T09:00:33.582 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:00:33.624 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:00:33.625 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T09:00:33.625 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:00:33.628 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): libgfortran-11.5.0-2.el9.x86_64.rpm 870 kB/s | 801 kB 00:00 2024-09-17T09:00:33.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-09-17T09:00:33.828 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 61 kB/s | 18 kB 00:00 2024-09-17T09:00:33.920 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 132 kB/s | 38 kB 00:00 2024-09-17T09:00:34.263 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:50 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:00:34.379 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 534 kB/s | 245 kB 00:00 2024-09-17T09:00:34.395 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:00:34.395 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:34.395 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T09:00:34.395 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:34.395 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:00:34.395 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 226 k 2024-09-17T09:00:34.395 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:34.396 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:00:34.396 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:34.396 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T09:00:34.396 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:34.396 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 226 k 2024-09-17T09:00:34.396 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 217 k 2024-09-17T09:00:34.396 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:00:34.429 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2024-09-17T09:00:34.546 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-17T09:00:34.609 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-18.2.4-855.g19e2a2be.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-09-17T09:00:34.609 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:34.610 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-09-17T09:00:34.610 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:00:34.615 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:00:34.615 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:00:34.618 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:00:34.618 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:00:34.692 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:00:35.114 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T09:00:35.206 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T09:00:35.346 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.3 MB/s | 4.9 MB 00:01 2024-09-17T09:00:35.565 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T09:00:35.824 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T09:00:35.824 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:35.824 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:00:35.824 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:35.824 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:35.824 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:00:36.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T09:00:36.288 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.7 MB/s | 5.1 MB 00:01 2024-09-17T09:00:36.617 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:00:36.749 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:00:36.749 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 143 k 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 143 k 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 395 k 2024-09-17T09:00:36.750 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:00:36.965 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-18.2.4-855.g19e2a2b 664 kB/s | 143 kB 00:00 2024-09-17T09:00:36.966 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:36.966 INFO:teuthology.orchestra.run.smithi173.stdout:Total 660 kB/s | 143 kB 00:00 2024-09-17T09:00:36.966 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:00:36.974 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:00:36.974 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:00:37.031 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:00:37.031 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:00:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:00:37.275 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T09:00:37.302 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T09:00:37.302 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:00:37.302 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T09:00:37.302 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T09:00:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:38.249 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:00:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:00:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:38.250 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:00:38.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-09-17T09:00:39.085 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:00:39.173 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:00:39.215 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:00:39.216 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T09:00:39.216 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:00:39.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T09:00:39.858 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:00:39.989 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:00:39.990 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:39.990 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T09:00:39.990 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:39.990 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:00:39.990 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 3.5 M 2024-09-17T09:00:39.990 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T09:00:39.990 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 24 k 2024-09-17T09:00:39.990 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 16 k 2024-09-17T09:00:39.990 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T09:00:39.991 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T09:00:39.991 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:39.991 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:00:39.991 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:39.991 INFO:teuthology.orchestra.run.smithi173.stdout:Install 5 Packages 2024-09-17T09:00:39.991 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:39.991 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.8 M 2024-09-17T09:00:39.992 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 87 M 2024-09-17T09:00:39.992 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:00:40.313 INFO:teuthology.orchestra.run.smithi173.stdout:(1/5): ceph-prometheus-alerts-18.2.4-855.g19e2a 98 kB/s | 16 kB 00:00 2024-09-17T09:00:40.337 INFO:teuthology.orchestra.run.smithi173.stdout:(2/5): ceph-grafana-dashboards-18.2.4-855.g19e2 127 kB/s | 24 kB 00:00 2024-09-17T09:00:40.387 INFO:teuthology.orchestra.run.smithi173.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 419 kB/s | 31 kB 00:00 2024-09-17T09:00:40.421 INFO:teuthology.orchestra.run.smithi173.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-09-17T09:00:40.529 INFO:teuthology.orchestra.run.smithi173.stdout:(5/5): ceph-mgr-dashboard-18.2.4-855.g19e2a2be. 9.2 MB/s | 3.5 MB 00:00 2024-09-17T09:00:40.533 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:40.533 INFO:teuthology.orchestra.run.smithi173.stdout:Total 7.0 MB/s | 3.8 MB 00:00 2024-09-17T09:00:40.533 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:00:40.540 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:00:40.540 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:00:40.620 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:00:40.620 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:00:40.897 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:00:40.993 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-17T09:00:41.026 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-17T09:00:41.046 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-17T09:00:42.086 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 4/5 2024-09-17T09:00:42.116 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-17T09:00:43.289 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-17T09:00:43.290 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 1/5 2024-09-17T09:00:43.290 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2/5 2024-09-17T09:00:43.290 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-17T09:00:43.290 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-17T09:00:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-17T09:00:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:43.560 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:00:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T09:00:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T09:00:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:43.561 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:00:43.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T09:00:44.367 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:00:44.499 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:00:44.500 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-09-17T09:00:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 7.4 M 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T09:00:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T09:00:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T09:00:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T09:00:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T09:00:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:44.502 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:00:44.502 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-09-17T09:00:44.502 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-09-17T09:00:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:44.503 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 38 M 2024-09-17T09:00:44.503 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 212 M 2024-09-17T09:00:44.503 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:00:44.953 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-17T09:00:45.053 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.8 MB/s | 801 kB 00:00 2024-09-17T09:00:45.237 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.4-85 16 MB/s | 7.4 MB 00:00 2024-09-17T09:00:45.579 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 52 kB/s | 33 kB 00:00 2024-09-17T09:00:45.796 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 33 kB/s | 18 kB 00:00 2024-09-17T09:00:46.129 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-09-17T09:00:46.155 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 67 kB/s | 38 kB 00:00 2024-09-17T09:00:46.946 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 300 kB/s | 245 kB 00:00 2024-09-17T09:00:47.022 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.1 MB/s | 4.9 MB 00:01 2024-09-17T09:00:47.339 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-09-17T09:00:47.730 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 320 kB/s | 250 kB 00:00 2024-09-17T09:00:48.472 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-17T09:00:48.474 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:48.474 INFO:teuthology.orchestra.run.smithi173.stdout:Total 9.6 MB/s | 38 MB 00:03 2024-09-17T09:00:48.840 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:00:48.866 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:00:48.866 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:00:49.389 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:00:49.390 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:00:50.148 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:00:50.228 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T09:00:50.278 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T09:00:50.378 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T09:00:50.421 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T09:00:50.617 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T09:00:50.645 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T09:00:50.730 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T09:00:51.379 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T09:00:51.499 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T09:00:52.735 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T09:00:53.769 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T09:00:53.795 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-17T09:00:55.093 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 810 kB/s | 16 MB 00:20 2024-09-17T09:00:55.094 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:55.095 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.7 MB/s | 38 MB 00:22 2024-09-17T09:00:55.451 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:00:55.478 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:00:55.479 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:00:55.994 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:00:55.994 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:00:56.478 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 1/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T09:00:56.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T09:00:56.748 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:00:56.828 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T09:00:56.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T09:00:56.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T09:00:56.884 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T09:00:56.884 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T09:00:56.884 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T09:00:56.884 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:56.884 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:00:56.884 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T09:00:56.992 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T09:00:57.034 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T09:00:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T09:00:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T09:00:57.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-09-17T09:00:57.327 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T09:00:57.870 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:13 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:00:58.002 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T09:00:58.003 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:00:58.004 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 49 k 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T09:00:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T09:00:58.006 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T09:00:58.006 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T09:00:58.006 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T09:00:58.006 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T09:00:58.006 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:58.006 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:00:58.006 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:00:58.006 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-09-17T09:00:58.006 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:00:58.007 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.2 M 2024-09-17T09:00:58.007 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-09-17T09:00:58.007 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:00:58.124 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T09:00:58.378 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 314 kB/s | 26 kB 00:00 2024-09-17T09:00:58.403 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 179 kB/s | 19 kB 00:00 2024-09-17T09:00:58.487 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-17T09:00:58.512 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): ceph-mgr-rook-18.2.4-855.g19e2a2be.el9. 228 kB/s | 49 kB 00:00 2024-09-17T09:00:58.537 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-17T09:00:58.571 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.3 MB/s | 279 kB 00:00 2024-09-17T09:00:58.596 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 918 kB/s | 54 kB 00:00 2024-09-17T09:00:58.630 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 350 kB/s | 32 kB 00:00 2024-09-17T09:00:58.663 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 156 kB/s | 14 kB 00:00 2024-09-17T09:00:58.697 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-17T09:00:58.739 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 10 MB/s | 1.0 MB 00:00 2024-09-17T09:00:58.764 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-09-17T09:00:58.789 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-google-auth-2.34.0-1.el9.noarc 1.2 MB/s | 220 kB 00:00 2024-09-17T09:00:58.792 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:00:58.792 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-09-17T09:00:58.857 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:00:58.873 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:00:58.873 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:00:58.960 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:00:58.967 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:00:59.415 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T09:00:59.468 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:00:59.642 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T09:00:59.698 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T09:00:59.754 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T09:00:59.792 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T09:00:59.832 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T09:00:59.934 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T09:01:00.036 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T09:01:00.084 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T09:01:00.441 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T09:01:00.469 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-17T09:01:00.792 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T09:01:00.849 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T09:01:00.882 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T09:01:00.930 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T09:01:00.952 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-17T09:01:01.812 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-17T09:01:01.812 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 1/13 2024-09-17T09:01:01.812 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T09:01:01.812 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T09:01:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T09:01:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T09:01:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T09:01:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T09:01:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T09:01:01.814 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T09:01:01.814 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T09:01:01.814 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T09:01:01.814 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T09:01:02.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T09:01:02.155 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T09:01:02.156 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T09:01:02.157 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T09:01:02.157 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T09:01:02.157 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T09:01:02.157 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:02.157 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:02.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T09:01:03.013 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:19 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:03.145 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:03.146 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T09:01:03.146 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 139 k 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-09-17T09:01:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:03.148 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 7.0 M 2024-09-17T09:01:03.148 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 33 M 2024-09-17T09:01:03.148 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:01:03.408 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-17T09:01:03.408 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 1/12 2024-09-17T09:01:03.408 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T09:01:03.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T09:01:03.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T09:01:03.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T09:01:03.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T09:01:03.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T09:01:03.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T09:01:03.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T09:01:03.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T09:01:03.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T09:01:03.476 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-09-17T09:01:03.543 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): ceph-mgr-cephadm-18.2.4-855.g19e2a2be.el 605 kB/s | 139 kB 00:00 2024-09-17T09:01:03.585 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-09-17T09:01:03.677 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-17T09:01:03.702 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 363 kB/s | 58 kB 00:00 2024-09-17T09:01:03.727 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 531 kB/s | 74 kB 00:00 2024-09-17T09:01:03.729 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:03.730 INFO:teuthology.orchestra.run.smithi173.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-17T09:01:03.801 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T09:01:03.804 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T09:01:03.805 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T09:01:03.805 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T09:01:03.805 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T09:01:03.805 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T09:01:03.805 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:03.805 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:03.812 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:01:03.812 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:01:03.912 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:01:03.913 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:01:04.224 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:01:04.245 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2024-09-17T09:01:04.392 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T09:01:04.445 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T09:01:04.838 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:32 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T09:01:04.969 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 49 k 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T09:01:04.971 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2024-09-17T09:01:04.972 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:04.973 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.2 M 2024-09-17T09:01:04.973 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-09-17T09:01:04.973 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:01:05.048 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T09:01:05.110 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T09:01:05.115 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 223 kB/s | 19 kB 00:00 2024-09-17T09:01:05.132 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-17T09:01:05.140 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 229 kB/s | 26 kB 00:00 2024-09-17T09:01:05.199 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): ceph-mgr-rook-18.2.4-855.g19e2a2be.el9. 290 kB/s | 49 kB 00:00 2024-09-17T09:01:05.234 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.8 MB/s | 222 kB 00:00 2024-09-17T09:01:05.260 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-17T09:01:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-17T09:01:05.326 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 479 kB/s | 32 kB 00:00 2024-09-17T09:01:05.351 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-17T09:01:05.379 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-17T09:01:05.410 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-17T09:01:05.436 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.7 MB/s | 90 kB 00:00 2024-09-17T09:01:05.460 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-09-17T09:01:05.494 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-17T09:01:05.497 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:05.497 INFO:teuthology.orchestra.run.smithi079.stdout:Total 4.2 MB/s | 2.2 MB 00:00 2024-09-17T09:01:05.560 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:01:05.575 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:01:05.575 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:01:05.662 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:01:05.663 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:01:05.935 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-17T09:01:05.935 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/6 2024-09-17T09:01:05.935 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T09:01:05.935 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T09:01:05.935 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T09:01:05.935 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T09:01:06.176 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:01:06.221 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T09:01:06.221 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:06.221 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:01:06.221 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:01:06.222 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T09:01:06.222 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T09:01:06.222 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T09:01:06.222 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T09:01:06.222 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T09:01:06.222 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:06.222 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:06.357 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T09:01:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T09:01:06.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-09-17T09:01:06.493 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T09:01:06.530 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T09:01:06.572 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T09:01:06.681 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T09:01:06.791 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T09:01:06.849 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T09:01:07.069 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:23 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:07.201 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:07.202 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:07.202 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T09:01:07.202 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:07.202 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:01:07.202 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 852 k 2024-09-17T09:01:07.202 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T09:01:07.202 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T09:01:07.202 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:07.203 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:01:07.203 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:07.203 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-09-17T09:01:07.203 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:07.203 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 932 k 2024-09-17T09:01:07.203 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.7 M 2024-09-17T09:01:07.203 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:01:07.520 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): ceph-fuse-18.2.4-855.g19e2a2be.el9.x86_6 3.1 MB/s | 852 kB 00:00 2024-09-17T09:01:07.560 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T09:01:07.614 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T09:01:07.647 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T09:01:07.704 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T09:01:07.734 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-17T09:01:07.887 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 126 kB/s | 80 kB 00:00 2024-09-17T09:01:07.888 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:07.888 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.3 MB/s | 932 kB 00:00 2024-09-17T09:01:07.900 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:01:07.923 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:01:07.923 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:01:08.003 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:01:08.003 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:01:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:01:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T09:01:08.386 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-17T09:01:08.522 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-17T09:01:08.522 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 1/13 2024-09-17T09:01:08.522 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T09:01:08.522 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T09:01:08.522 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T09:01:08.522 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T09:01:08.522 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T09:01:08.522 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T09:01:08.523 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T09:01:08.523 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T09:01:08.523 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T09:01:08.523 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T09:01:08.523 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T09:01:08.864 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T09:01:08.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T09:01:08.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T09:01:08.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T09:01:08.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T09:01:08.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T09:01:08.866 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:08.866 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:09.071 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-17T09:01:09.071 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/2 2024-09-17T09:01:09.119 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T09:01:09.390 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T09:01:09.391 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:09.391 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:01:09.391 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T09:01:09.391 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:09.391 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:09.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-volume 2024-09-17T09:01:09.694 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:09.825 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:09.826 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:09.826 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T09:01:09.826 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:09.826 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:01:09.826 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 139 k 2024-09-17T09:01:09.826 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-09-17T09:01:09.827 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:09.828 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 7.0 M 2024-09-17T09:01:09.828 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 33 M 2024-09-17T09:01:09.828 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:01:10.058 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-17T09:01:10.125 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): ceph-mgr-cephadm-18.2.4-855.g19e2a2be.el 626 kB/s | 139 kB 00:00 2024-09-17T09:01:10.167 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-09-17T09:01:10.201 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.3 MB/s | 74 kB 00:00 2024-09-17T09:01:10.226 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 574 kB/s | 58 kB 00:00 2024-09-17T09:01:10.286 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:26 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:10.301 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-17T09:01:10.304 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:10.304 INFO:teuthology.orchestra.run.smithi079.stdout:Total 15 MB/s | 7.0 MB 00:00 2024-09-17T09:01:10.373 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:01:10.384 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:01:10.385 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:01:10.421 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 264 k 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:01:10.422 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:10.423 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-09-17T09:01:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:10.423 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 342 k 2024-09-17T09:01:10.423 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 1.5 M 2024-09-17T09:01:10.423 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:01:10.484 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:01:10.485 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:01:10.659 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): ceph-volume-18.2.4-855.g19e2a2be.el9.noa 1.2 MB/s | 264 kB 00:00 2024-09-17T09:01:10.692 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 312 kB/s | 77 kB 00:00 2024-09-17T09:01:10.693 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:10.693 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.2 MB/s | 342 kB 00:00 2024-09-17T09:01:10.705 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:01:10.715 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:01:10.715 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:01:10.754 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:01:10.754 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:01:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:01:10.911 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:01:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T09:01:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T09:01:11.091 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-17T09:01:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-17T09:01:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-17T09:01:11.137 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:01:11.137 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T09:01:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:11.534 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T09:01:11.624 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T09:01:11.693 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T09:01:11.716 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-17T09:01:11.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 1/2 2024-09-17T09:01:12.197 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-17T09:01:12.197 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:12.197 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:01:12.197 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:01:12.198 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T09:01:12.198 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:12.198 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:12.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-09-17T09:01:12.551 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-17T09:01:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/6 2024-09-17T09:01:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T09:01:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T09:01:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T09:01:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T09:01:12.837 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T09:01:12.837 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:12.837 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:12.837 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:01:12.838 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T09:01:12.838 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T09:01:12.838 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T09:01:12.838 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T09:01:12.838 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T09:01:12.838 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:12.838 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:13.116 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2024-09-17T09:01:13.126 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:29 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:13.260 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:13.261 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-09-17T09:01:13.261 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:13.261 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:01:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 126 k 2024-09-17T09:01:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:13.261 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:01:13.261 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:13.262 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T09:01:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:13.262 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 126 k 2024-09-17T09:01:13.262 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 456 k 2024-09-17T09:01:13.262 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:01:13.464 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-18.2.4-855.g19e2a2be.el9.x86_64. 626 kB/s | 126 kB 00:00 2024-09-17T09:01:13.465 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:13.465 INFO:teuthology.orchestra.run.smithi173.stdout:Total 622 kB/s | 126 kB 00:00 2024-09-17T09:01:13.465 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:01:13.470 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:01:13.471 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:01:13.534 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:01:13.534 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:01:13.684 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:01:13.690 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:13.795 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:13.825 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 852 k 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-09-17T09:01:13.826 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:13.827 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 932 k 2024-09-17T09:01:13.827 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 2.7 M 2024-09-17T09:01:13.827 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:01:14.097 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 531 kB/s | 80 kB 00:00 2024-09-17T09:01:14.239 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-fuse-18.2.4-855.g19e2a2be.el9.x86_6 2.8 MB/s | 852 kB 00:00 2024-09-17T09:01:14.239 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:14.240 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.2 MB/s | 932 kB 00:00 2024-09-17T09:01:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:14.251 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:01:14.273 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:01:14.273 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:01:14.353 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:01:14.353 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:01:14.519 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:14.519 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:14.519 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:01:14.519 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:14.520 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:14.520 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:14.539 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:01:14.679 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T09:01:14.755 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-17T09:01:14.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-09-17T09:01:15.374 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:31 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:15.416 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-17T09:01:15.416 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/2 2024-09-17T09:01:15.468 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:15.514 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:15.515 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T09:01:15.515 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:15.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-09-17T09:01:15.688 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T09:01:15.688 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:15.688 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:15.688 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T09:01:15.688 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:15.688 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:15.987 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-volume 2024-09-17T09:01:16.162 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:32 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:16.294 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:16.295 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-09-17T09:01:16.295 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:16.295 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:01:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 31 k 2024-09-17T09:01:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:16.296 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:01:16.296 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:16.296 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T09:01:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:16.296 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 31 k 2024-09-17T09:01:16.296 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 139 k 2024-09-17T09:01:16.296 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:01:16.452 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-18.2.4-855.g19e2a2be.el9.x86_64 200 kB/s | 31 kB 00:00 2024-09-17T09:01:16.453 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:16.453 INFO:teuthology.orchestra.run.smithi173.stdout:Total 197 kB/s | 31 kB 00:00 2024-09-17T09:01:16.453 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:01:16.456 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:01:16.456 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:01:16.469 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:01:16.470 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:01:16.555 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:01:16.582 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:16.719 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:16.720 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:16.720 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-17T09:01:16.720 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:16.720 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:01:16.720 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 264 k 2024-09-17T09:01:16.720 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-17T09:01:16.720 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T09:01:16.721 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:16.721 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:01:16.721 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:16.721 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-09-17T09:01:16.721 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:16.721 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 342 k 2024-09-17T09:01:16.721 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 1.5 M 2024-09-17T09:01:16.722 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:01:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:17.147 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 525 kB/s | 77 kB 00:00 2024-09-17T09:01:17.230 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-volume-18.2.4-855.g19e2a2be.el9.noa 1.1 MB/s | 264 kB 00:00 2024-09-17T09:01:17.230 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:17.231 INFO:teuthology.orchestra.run.smithi079.stdout:Total 669 kB/s | 342 kB 00:00 2024-09-17T09:01:17.243 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:01:17.251 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:17.251 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:17.251 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:01:17.251 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:17.251 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:17.251 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:17.252 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:01:17.252 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:01:17.284 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:01:17.285 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:01:17.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-09-17T09:01:17.434 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:01:17.613 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-17T09:01:17.658 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-17T09:01:17.674 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-17T09:01:17.674 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:01:17.674 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T09:01:17.674 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:18.001 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:18.089 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:18.132 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:18.133 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T09:01:18.133 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:18.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-09-17T09:01:18.398 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 1/2 2024-09-17T09:01:18.728 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-17T09:01:18.729 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:18.729 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:18.729 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T09:01:18.729 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T09:01:18.729 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:18.729 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:18.781 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:18.870 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:18.913 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:18.914 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T09:01:18.914 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:18.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-09-17T09:01:19.065 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2024-09-17T09:01:19.569 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:35 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:19.655 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:19.661 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:19.705 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:19.706 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T09:01:19.706 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:19.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-09-17T09:01:19.787 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:19.787 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:19.788 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-09-17T09:01:19.788 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:19.788 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:01:19.788 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 126 k 2024-09-17T09:01:19.788 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:19.788 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:01:19.788 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:19.788 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-17T09:01:19.788 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:19.790 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 126 k 2024-09-17T09:01:19.790 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 456 k 2024-09-17T09:01:19.790 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:01:20.006 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-18.2.4-855.g19e2a2be.el9.x86_64. 583 kB/s | 126 kB 00:00 2024-09-17T09:01:20.007 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:20.007 INFO:teuthology.orchestra.run.smithi079.stdout:Total 579 kB/s | 126 kB 00:00 2024-09-17T09:01:20.007 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:01:20.012 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:01:20.013 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:01:20.078 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:01:20.079 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:01:20.229 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:01:20.337 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:20.352 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:20.443 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:20.487 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:20.488 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T09:01:20.489 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:20.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-09-17T09:01:20.789 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:21.044 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:21.045 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:21.045 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:21.045 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:21.045 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:21.045 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:21.125 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:21.214 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:21.257 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:21.258 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T09:01:21.258 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:21.309 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2024-09-17T09:01:21.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-09-17T09:01:21.901 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:21.906 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:49 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:21.993 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:22.000 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:22.037 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:22.038 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T09:01:22.038 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:22.046 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:22.047 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-17T09:01:22.047 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:22.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-09-17T09:01:22.121 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2024-09-17T09:01:22.688 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:38 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:22.703 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:50 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:22.820 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:22.821 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T09:01:22.821 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:22.821 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:01:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 87 k 2024-09-17T09:01:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:22.821 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:01:22.821 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:22.822 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T09:01:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:22.822 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 87 k 2024-09-17T09:01:22.822 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 227 k 2024-09-17T09:01:22.822 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:01:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:22.835 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-09-17T09:01:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:01:22.836 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 31 k 2024-09-17T09:01:22.836 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:01:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-17T09:01:22.836 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:22.837 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 31 k 2024-09-17T09:01:22.837 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 139 k 2024-09-17T09:01:22.837 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:01:22.997 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-18.2.4-855.g19e2a2be.el9.x86_64 194 kB/s | 31 kB 00:00 2024-09-17T09:01:22.997 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:22.998 INFO:teuthology.orchestra.run.smithi079.stdout:Total 192 kB/s | 31 kB 00:00 2024-09-17T09:01:22.998 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:01:23.001 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:01:23.001 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:01:23.020 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:01:23.020 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:01:23.021 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-18.2.4-855.g19e2a2be.el9.x86_64.rpm 435 kB/s | 87 kB 00:00 2024-09-17T09:01:23.022 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:23.022 INFO:teuthology.orchestra.run.smithi173.stdout:Total 432 kB/s | 87 kB 00:00 2024-09-17T09:01:23.022 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:01:23.029 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:01:23.029 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:01:23.092 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:01:23.093 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:01:23.112 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:01:23.236 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:23.239 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:01:23.361 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:23.591 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:23.756 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:23.818 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:23.819 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:23.819 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:23.819 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:23.819 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:23.819 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:24.002 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2024-09-17T09:01:24.006 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:24.006 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:24.006 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:01:24.006 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:24.007 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:24.007 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:24.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-09-17T09:01:24.582 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:52 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:24.670 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:24.713 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:24.714 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-17T09:01:24.714 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:24.794 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2024-09-17T09:01:24.831 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:24.973 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:24.973 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.0 M 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 12 M 2024-09-17T09:01:24.974 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:01:25.342 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-18.2.4-855.g19e2a2be.el9.x86_64.rpm 8.2 MB/s | 3.0 MB 00:00 2024-09-17T09:01:25.342 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:25.342 INFO:teuthology.orchestra.run.smithi173.stdout:Total 8.1 MB/s | 3.0 MB 00:00 2024-09-17T09:01:25.343 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:01:25.351 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:01:25.352 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:01:25.368 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:53 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:25.434 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:01:25.434 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:01:25.457 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:25.500 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:25.501 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-17T09:01:25.501 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:25.579 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2024-09-17T09:01:25.856 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:01:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:25.951 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:01:25.951 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T09:01:25.951 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T09:01:25.951 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T09:01:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:26.152 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:54 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:26.245 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:26.288 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:26.289 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-17T09:01:26.289 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:26.370 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2024-09-17T09:01:26.951 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:54 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:26.991 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:26.991 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:26.991 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:01:26.991 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:26.991 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:26.992 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:27.042 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:27.086 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:27.087 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-17T09:01:27.087 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:27.173 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2024-09-17T09:01:27.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-09-17T09:01:27.750 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:55 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:27.834 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 08:59:44 AM UTC. 2024-09-17T09:01:27.839 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:27.881 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:27.882 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-17T09:01:27.882 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:27.970 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T09:01:27.971 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:27.971 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T09:01:27.971 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:27.971 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T09:01:27.971 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 172 k 2024-09-17T09:01:27.972 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:27.972 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T09:01:27.972 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T09:01:27.972 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T09:01:27.972 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:27.972 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 172 k 2024-09-17T09:01:27.972 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 494 k 2024-09-17T09:01:27.972 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T09:01:27.973 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2024-09-17T09:01:28.178 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-18.2.4-855.g19e2a2be.el9.x86_64.rpm 836 kB/s | 172 kB 00:00 2024-09-17T09:01:28.179 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:28.179 INFO:teuthology.orchestra.run.smithi173.stdout:Total 830 kB/s | 172 kB 00:00 2024-09-17T09:01:28.179 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T09:01:28.187 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T09:01:28.187 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T09:01:28.250 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T09:01:28.250 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T09:01:28.403 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T09:01:28.493 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:28.550 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:56 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:28.642 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T09:01:28.686 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:28.687 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-17T09:01:28.688 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:28.762 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2024-09-17T09:01:28.956 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:29.215 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:29.215 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:29.215 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T09:01:29.215 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:29.215 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:01:29.215 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T09:01:29.338 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:57 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:29.469 DEBUG:teuthology.parallel:result is None 2024-09-17T09:01:29.471 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:29.471 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:29.471 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-09-17T09:01:29.471 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:29.471 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:01:29.471 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 87 k 2024-09-17T09:01:29.472 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:29.472 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:01:29.472 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:29.472 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-17T09:01:29.472 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:29.473 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 87 k 2024-09-17T09:01:29.473 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 227 k 2024-09-17T09:01:29.473 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:01:29.658 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-18.2.4-855.g19e2a2be.el9.x86_64.rpm 468 kB/s | 87 kB 00:00 2024-09-17T09:01:29.658 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:29.659 INFO:teuthology.orchestra.run.smithi079.stdout:Total 464 kB/s | 87 kB 00:00 2024-09-17T09:01:29.659 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:01:29.666 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:01:29.666 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:01:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:01:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:01:29.901 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:01:29.996 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:30.366 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:30.612 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:30.612 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:30.612 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:30.612 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:30.612 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:30.612 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:30.858 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2024-09-17T09:01:31.494 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:59 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:31.638 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:31.639 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:31.639 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-09-17T09:01:31.639 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:31.639 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:01:31.639 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-17T09:01:31.639 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:31.640 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:01:31.640 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:31.640 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-17T09:01:31.640 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:31.640 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.0 M 2024-09-17T09:01:31.640 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 12 M 2024-09-17T09:01:31.640 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:01:31.987 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-18.2.4-855.g19e2a2be.el9.x86_64.rpm 8.6 MB/s | 3.0 MB 00:00 2024-09-17T09:01:31.987 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:31.988 INFO:teuthology.orchestra.run.smithi079.stdout:Total 8.6 MB/s | 3.0 MB 00:00 2024-09-17T09:01:31.988 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:01:31.997 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:01:31.997 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:01:32.079 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:01:32.080 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:01:32.495 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:01:32.563 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:32.590 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:32.590 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T09:01:32.590 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T09:01:32.590 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T09:01:32.590 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T09:01:32.590 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:33.725 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:33.725 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:33.726 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:33.726 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:33.726 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:33.726 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:33.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2024-09-17T09:01:34.584 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:02 ago on Tue 17 Sep 2024 08:59:32 AM UTC. 2024-09-17T09:01:34.716 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-17T09:01:34.717 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:34.717 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-09-17T09:01:34.717 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:34.717 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-17T09:01:34.717 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 172 k 2024-09-17T09:01:34.717 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:34.717 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-17T09:01:34.717 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-17T09:01:34.717 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-17T09:01:34.718 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:34.718 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 172 k 2024-09-17T09:01:34.718 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 494 k 2024-09-17T09:01:34.718 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-17T09:01:34.924 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-18.2.4-855.g19e2a2be.el9.x86_64.rpm 834 kB/s | 172 kB 00:00 2024-09-17T09:01:34.924 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:01:34.925 INFO:teuthology.orchestra.run.smithi079.stdout:Total 829 kB/s | 172 kB 00:00 2024-09-17T09:01:34.925 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-17T09:01:34.932 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-17T09:01:34.932 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-17T09:01:34.997 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-17T09:01:34.997 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-17T09:01:35.159 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-17T09:01:35.263 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:35.727 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:35.984 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T09:01:35.984 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:35.985 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-17T09:01:35.985 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T09:01:35.985 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:01:35.985 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-17T09:01:36.218 DEBUG:teuthology.parallel:result is None 2024-09-17T09:01:36.219 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:01:36.366 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T09:01:36.388 INFO:teuthology.orchestra.run.smithi079.stdout:18.2.4-855.g19e2a2be.el9 2024-09-17T09:01:36.388 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855.g19e2a2be.el9 2024-09-17T09:01:36.388 INFO:teuthology.task.install:The correct ceph version 18.2.4-855.g19e2a2be is installed. 2024-09-17T09:01:36.391 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:01:36.535 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T09:01:36.562 INFO:teuthology.orchestra.run.smithi173.stdout:18.2.4-855.g19e2a2be.el9 2024-09-17T09:01:36.562 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855.g19e2a2be.el9 2024-09-17T09:01:36.562 INFO:teuthology.task.install:The correct ceph version 18.2.4-855.g19e2a2be is installed. 2024-09-17T09:01:36.564 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T09:01:36.564 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:01:36.564 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T09:01:36.590 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:01:36.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T09:01:36.633 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T09:01:36.633 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:01:36.633 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T09:01:36.659 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T09:01:36.725 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:01:36.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T09:01:36.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T09:01:36.818 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T09:01:36.819 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:01:36.819 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T09:01:36.844 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T09:01:36.909 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:01:36.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T09:01:36.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T09:01:37.001 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T09:01:37.002 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:01:37.002 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T09:01:37.048 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T09:01:37.113 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:01:37.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T09:01:37.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T09:01:37.205 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-17T09:01:37.305 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'cephadm_mode': 'root'} 2024-09-17T09:01:37.306 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:01:37.306 INFO:tasks.cephadm:Cluster fsid is 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:01:37.306 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-17T09:01:37.307 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.79', 'mon.c': '[v2:172.21.15.79:3301,v1:172.21.15.79:6790]', 'mon.b': '172.21.15.173'} 2024-09-17T09:01:37.307 INFO:tasks.cephadm:First mon is mon.a on smithi079 2024-09-17T09:01:37.307 INFO:tasks.cephadm:First mgr is y 2024-09-17T09:01:37.307 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-17T09:01:37.307 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2024-09-17T09:01:37.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-09-17T09:01:37.359 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-17T09:01:37.359 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:01:37.501 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56094/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.4-855-g19e2a2be', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.4-855.g19e2a2be'}, 'url': 'https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-16 16:29:21.739533', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'reef', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-17T09:01:37.646 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:01:37.647 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm 2024-09-17T09:01:37.648 DEBUG:teuthology.orchestra.run.smithi079:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T09:01:37.973 INFO:teuthology.orchestra.run.smithi079.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 17 09:01 /home/ubuntu/cephtest/cephadm 2024-09-17T09:01:37.974 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T09:01:38.335 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 17 09:01 /home/ubuntu/cephtest/cephadm 2024-09-17T09:01:38.335 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T09:01:38.355 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T09:01:38.386 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 on all hosts... 2024-09-17T09:01:38.387 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 pull 2024-09-17T09:01:38.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 pull 2024-09-17T09:01:39.984 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9... 2024-09-17T09:01:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9... 2024-09-17T09:02:34.612 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T09:02:34.613 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)", 2024-09-17T09:02:34.613 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89", 2024-09-17T09:02:34.613 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-09-17T09:02:34.613 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9" 2024-09-17T09:02:34.613 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T09:02:34.614 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T09:02:34.617 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-17T09:02:34.617 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)", 2024-09-17T09:02:34.617 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89", 2024-09-17T09:02:34.617 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2024-09-17T09:02:34.617 INFO:teuthology.orchestra.run.smithi079.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9" 2024-09-17T09:02:34.617 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-17T09:02:34.617 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-17T09:02:34.636 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2024-09-17T09:02:34.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-09-17T09:02:34.722 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2024-09-17T09:02:34.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-09-17T09:02:34.787 INFO:tasks.cephadm:Writing seed config... 2024-09-17T09:02:34.788 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-17T09:02:34.788 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-17T09:02:34.788 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-17T09:02:34.789 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-09-17T09:02:34.789 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-09-17T09:02:34.789 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-17T09:02:34.789 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-17T09:02:34.789 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-17T09:02:34.789 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-17T09:02:34.789 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-17T09:02:34.789 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-17T09:02:34.789 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-17T09:02:34.790 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:02:34.790 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-17T09:02:34.805 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 721af936-74d3-11ef-bceb-c7b262605968 mon election default strategy = 3 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-17T09:02:34.806 DEBUG:teuthology.orchestra.run.smithi079:mon.a> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.a.service 2024-09-17T09:02:34.848 DEBUG:teuthology.orchestra.run.smithi079:mgr.y> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y.service 2024-09-17T09:02:34.891 INFO:tasks.cephadm:Bootstrapping... 2024-09-17T09:02:34.892 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -v bootstrap --fsid 721af936-74d3-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.79 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:02:35.043 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:02:35.043 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', '-v', 'bootstrap', '--fsid', '721af936-74d3-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.79', '--skip-admin-label'] 2024-09-17T09:02:35.067 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5.2.2 2024-09-17T09:02:35.067 INFO:teuthology.orchestra.run.smithi079.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-17T09:02:35.067 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying podman|docker is present... 2024-09-17T09:02:35.089 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5.2.2 2024-09-17T09:02:35.089 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying lvm2 is present... 2024-09-17T09:02:35.089 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying time synchronization is in place... 2024-09-17T09:02:35.096 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-17T09:02:35.096 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T09:02:35.102 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-17T09:02:35.103 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-17T09:02:35.109 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-09-17T09:02:35.116 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-09-17T09:02:35.116 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-09-17T09:02:35.116 INFO:teuthology.orchestra.run.smithi079.stdout:Repeating the final host check... 2024-09-17T09:02:35.137 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5.2.2 2024-09-17T09:02:35.137 INFO:teuthology.orchestra.run.smithi079.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-17T09:02:35.137 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl is present 2024-09-17T09:02:35.137 INFO:teuthology.orchestra.run.smithi079.stdout:lvcreate is present 2024-09-17T09:02:35.144 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-17T09:02:35.144 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T09:02:35.150 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-17T09:02:35.150 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-17T09:02:35.156 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-09-17T09:02:35.163 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-09-17T09:02:35.163 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-09-17T09:02:35.163 INFO:teuthology.orchestra.run.smithi079.stdout:Host looks OK 2024-09-17T09:02:35.163 INFO:teuthology.orchestra.run.smithi079.stdout:Cluster fsid: 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:02:35.163 INFO:teuthology.orchestra.run.smithi079.stdout:Acquiring lock 139749014695504 on /run/cephadm/721af936-74d3-11ef-bceb-c7b262605968.lock 2024-09-17T09:02:35.163 INFO:teuthology.orchestra.run.smithi079.stdout:Lock 139749014695504 acquired on /run/cephadm/721af936-74d3-11ef-bceb-c7b262605968.lock 2024-09-17T09:02:35.163 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 3300 ... 2024-09-17T09:02:35.164 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 6789 ... 2024-09-17T09:02:35.164 INFO:teuthology.orchestra.run.smithi079.stdout:Base mon IP(s) is [172.21.15.79:3300, 172.21.15.79:6789], mon addrv is [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-09-17T09:02:35.167 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.79 metric 100 2024-09-17T09:02:35.167 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.79 metric 100 2024-09-17T09:02:35.169 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-17T09:02:35.169 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-17T09:02:35.169 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-17T09:02:35.172 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-17T09:02:35.172 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-17T09:02:35.172 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-17T09:02:35.172 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-17T09:02:35.172 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce87/64 scope link noprefixroute 2024-09-17T09:02:35.172 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-17T09:02:35.173 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-09-17T09:02:35.173 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-09-17T09:02:35.173 INFO:teuthology.orchestra.run.smithi079.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-17T09:02:35.173 INFO:teuthology.orchestra.run.smithi079.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-17T09:02:35.174 INFO:teuthology.orchestra.run.smithi079.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9... 2024-09-17T09:02:35.604 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 2024-09-17T09:02:35.604 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9... 2024-09-17T09:02:35.604 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Getting image source signatures 2024-09-17T09:02:35.604 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:d8a707c21dc06a64315160cb2d8e524fa91943a344731b2a930ee3a716bebbdd 2024-09-17T09:02:35.604 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-17T09:02:35.604 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying config sha256:980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 2024-09-17T09:02:35.604 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-17T09:02:36.458 INFO:teuthology.orchestra.run.smithi079.stdout:ceph: stdout ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable) 2024-09-17T09:02:36.458 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph version: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable) 2024-09-17T09:02:36.458 INFO:teuthology.orchestra.run.smithi079.stdout:Extracting ceph user uid/gid from container image... 2024-09-17T09:02:37.339 INFO:teuthology.orchestra.run.smithi079.stdout:stat: stdout 167 167 2024-09-17T09:02:37.339 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial keys... 2024-09-17T09:02:38.371 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQCtRelmYnocKRAAcZnXwHdn3FfbVOdckbsa4Q== 2024-09-17T09:02:39.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQCuRelmLXGKKhAAeZPBVKEJvMCOgBTdPrq8qw== 2024-09-17T09:02:40.390 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQCvRelmdsJNKhAAMDTo8eicjAKNftsPvDX3rg== 2024-09-17T09:02:40.391 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial monmap... 2024-09-17T09:02:41.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T09:02:41.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-17T09:02:41.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:02:41.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T09:02:41.405 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool for a [v2:172.21.15.79:3300,v1:172.21.15.79:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T09:02:41.405 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = pacific 2024-09-17T09:02:41.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: set fsid to 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:02:41.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T09:02:41.406 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:02:41.406 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mon... 2024-09-17T09:02:42.419 INFO:teuthology.orchestra.run.smithi079.stdout:create mon.a on 2024-09-17T09:02:42.606 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-17T09:02:42.748 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-17T09:02:42.922 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-721af936-74d3-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-721af936-74d3-11ef-bceb-c7b262605968.target. 2024-09-17T09:02:42.923 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-721af936-74d3-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-721af936-74d3-11ef-bceb-c7b262605968.target. 2024-09-17T09:02:43.159 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.a 2024-09-17T09:02:43.159 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-17T09:02:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-721af936-74d3-11ef-bceb-c7b262605968.target.wants/ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-721af936-74d3-11ef-bceb-c7b262605968@.service. 2024-09-17T09:02:43.829 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:43 smithi079 ceph-mon[29111]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T09:02:43.829 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:43 smithi079 bash[29073]: 6a9629f12ff49156e72d97df09edf8f0c2fae184f80bbdbfa8705a6eea7891ac 2024-09-17T09:02:43.842 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T09:02:43.842 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-09-17T09:02:43.849 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T09:02:43.849 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-17T09:02:43.849 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-09-17T09:02:43.850 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-09-17T09:02:43.850 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon to start... 2024-09-17T09:02:43.850 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon... 2024-09-17T09:02:44.086 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:43 smithi079 systemd[1]: Started Ceph mon.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:02:44.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout cluster: 2024-09-17T09:02:44.881 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout id: 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:02:44.881 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout services: 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.558124s) 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout data: 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pgs: 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:02:44.882 INFO:teuthology.orchestra.run.smithi079.stdout:mon is available 2024-09-17T09:02:44.883 INFO:teuthology.orchestra.run.smithi079.stdout:Assimilating anything we can from ceph.conf... 2024-09-17T09:02:45.001 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:44 smithi079 ceph-mon[29111]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T09:02:45.002 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:44 smithi079 ceph-mon[29111]: monmap e1: 1 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:02:45.002 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:44 smithi079 ceph-mon[29111]: fsmap 2024-09-17T09:02:45.002 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:44 smithi079 ceph-mon[29111]: osdmap e1: 0 total, 0 up, 0 in 2024-09-17T09:02:45.002 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:44 smithi079 ceph-mon[29111]: mgrmap e1: no daemons active 2024-09-17T09:02:45.002 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:44 smithi079 ceph-mon[29111]: from='client.? 172.21.15.79:0/1803272213' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-17T09:02:45.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:02:45.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-09-17T09:02:45.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:02:45.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-09-17T09:02:45.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-17T09:02:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:Generating new minimal ceph.conf... 2024-09-17T09:02:46.034 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:45 smithi079 ceph-mon[29111]: from='client.? 172.21.15.79:0/843578613' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-17T09:02:46.034 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:45 smithi079 ceph-mon[29111]: from='client.? 172.21.15.79:0/843578613' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-17T09:02:46.959 INFO:teuthology.orchestra.run.smithi079.stdout:Restarting the monitor... 2024-09-17T09:02:47.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:46 smithi079 ceph-mon[29111]: from='client.? 172.21.15.79:0/1533843180' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:02:47.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:46 smithi079 systemd[1]: Stopping Ceph mon.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:02:47.534 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a[29088]: 2024-09-17T09:02:47.235+0000 7f6a704ba640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:02:47.534 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a[29088]: 2024-09-17T09:02:47.235+0000 7f6a704ba640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-17T09:02:47.535 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:47 smithi079 podman[29428]: 2024-09-17 09:02:47.308328189 +0000 UTC m=+0.188700198 container died 6a9629f12ff49156e72d97df09edf8f0c2fae184f80bbdbfa8705a6eea7891ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T09:02:47.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:47 smithi079 podman[29428]: 2024-09-17 09:02:47.617078612 +0000 UTC m=+0.497450621 container cleanup 6a9629f12ff49156e72d97df09edf8f0c2fae184f80bbdbfa8705a6eea7891ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:02:47.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:47 smithi079 bash[29428]: ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a 2024-09-17T09:02:48.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:47 smithi079 podman[29445]: 2024-09-17 09:02:47.79309803 +0000 UTC m=+0.479480344 container remove 6a9629f12ff49156e72d97df09edf8f0c2fae184f80bbdbfa8705a6eea7891ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T09:02:48.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:47 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T09:02:48.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:47 smithi079 systemd[1]: Stopped Ceph mon.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:02:48.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:47 smithi079 systemd[1]: Starting Ceph mon.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:02:48.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 podman[29539]: 2024-09-17 09:02:48.144063882 +0000 UTC m=+0.106451361 container create add4676afaeed12f8a9808966d369f6c420791ee4ac18b758b05b11527f97597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T09:02:48.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 podman[29539]: 2024-09-17 09:02:48.053155161 +0000 UTC m=+0.015542584 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:02:48.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 podman[29539]: 2024-09-17 09:02:48.339994066 +0000 UTC m=+0.302381483 container init add4676afaeed12f8a9808966d369f6c420791ee4ac18b758b05b11527f97597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-17T09:02:48.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 podman[29539]: 2024-09-17 09:02:48.344726112 +0000 UTC m=+0.307113526 container start add4676afaeed12f8a9808966d369f6c420791ee4ac18b758b05b11527f97597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:02:48.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T09:02:48.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 7 2024-09-17T09:02:48.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: pidfile_write: ignore empty --pid-file 2024-09-17T09:02:48.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: load: jerasure load: lrc 2024-09-17T09:02:48.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: RocksDB version: 7.9.2 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Git sha 0 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: DB SUMMARY 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: DB Session ID: YYG6KFYZ5C57YVJO5FF7 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: CURRENT file: CURRENT 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 85014 ; 2024-09-17T09:02:48.425 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.error_if_exists: 0 2024-09-17T09:02:48.426 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.create_if_missing: 0 2024-09-17T09:02:48.426 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.paranoid_checks: 1 2024-09-17T09:02:48.426 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T09:02:48.426 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T09:02:48.426 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T09:02:48.426 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.env: 0x55bbd333ebc0 2024-09-17T09:02:48.426 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T09:02:48.426 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.info_log: 0x55bbd4942be0 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.statistics: (nil) 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.use_fsync: 0 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_log_file_size: 0 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.allow_fallocate: 1 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:02:48.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.use_direct_reads: 0 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.db_log_dir: 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.wal_dir: 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:02:48.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.write_buffer_manager: 0x55bbd4950320 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.unordered_write: 0 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:02:48.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.row_cache: None 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.wal_filter: None 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.two_write_queues: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.wal_compression: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.atomic_flush: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.log_readahead_size: 0 2024-09-17T09:02:48.430 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_background_jobs: 2 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_background_compactions: -1 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_subcompactions: 1 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:02:48.431 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_open_files: -1 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_background_flushes: -1 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Compression algorithms supported: 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: kZSTD supported: 0 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: kXpressCompression supported: 0 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: kBZip2Compression supported: 0 2024-09-17T09:02:48.432 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: kLZ4Compression supported: 1 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: kZlibCompression supported: 1 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: kSnappyCompression supported: 1 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.merge_operator: 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_filter: None 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T09:02:48.433 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bbd49427e0) 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: cache_index_and_filter_blocks: 1 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: pin_top_level_index_and_filter: 1 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_type: 0 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: data_block_index_type: 0 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_shortening: 1 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: checksum: 4 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: no_block_cache: 0 2024-09-17T09:02:48.434 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache: 0x55bbd4937350 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_name: BinnedLRUCache 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_options: 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: capacity : 536870912 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: num_shard_bits : 4 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: strict_capacity_limit : 0 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: high_pri_pool_ratio: 0.000 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_compressed: (nil) 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: persistent_cache: (nil) 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_size: 4096 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_size_deviation: 10 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_restart_interval: 16 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_block_restart_interval: 1 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: metadata_block_size: 4096 2024-09-17T09:02:48.435 INFO:journalctl@ceph.mon.a.smithi079.stdout: partition_filters: 0 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: use_delta_encoding: 1 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: filter_policy: bloomfilter 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: whole_key_filtering: 1 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: verify_compression: 0 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: read_amp_bytes_per_bit: 0 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: format_version: 5 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: enable_index_compression: 1 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_align: 0 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: max_auto_readahead_size: 262144 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: prepopulate_block_cache: 0 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: initial_auto_readahead_size: 8192 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:02:48.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression: NoCompression 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.num_levels: 7 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:02:48.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:02:48.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:02:48.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:02:48.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:02:48.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:02:48.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:02:48.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.inplace_update_support: 0 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.bloom_locality: 0 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.max_successive_merges: 0 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:02:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.ttl: 2592000 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.enable_blob_files: false 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.min_blob_size: 0 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T09:02:48.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2e8d8748-b8bc-4acb-8dc1-3ca6a4e81fcf 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726563768370349, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726563768371266, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81573, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 256, "table_properties": {"data_size": 79734, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10217, "raw_average_key_size": 47, "raw_value_size": 73965, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 215, "num_filter_entries": 215, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726563768, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2e8d8748-b8bc-4acb-8dc1-3ca6a4e81fcf", "db_session_id": "YYG6KFYZ5C57YVJO5FF7", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726563768371361, "job": 1, "event": "recovery_finished"} 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bbd4962e00 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: DB pointer 0x55bbd4a56000 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T09:02:48.444 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** DB Stats ** 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** Compaction Stats [default] ** 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: L0 2/0 81.52 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Sum 2/0 81.52 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T09:02:48.445 INFO:journalctl@ceph.mon.a.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** Compaction Stats [default] ** 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative compaction: 0.00 GB write, 25.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:02:48.446 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval compaction: 0.00 GB write, 25.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: Block cache BinnedLRUCache@0x55bbd4937350#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: starting mon.a rank 0 at public addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] at bind addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mon.a@-1(???) e1 preinit fsid 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mon.a@-1(???).mds e1 new map 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mon.a@-1(???).mds e1 print_map 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: e1 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: legacy client fscid: -1 2024-09-17T09:02:48.447 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-17T09:02:48.448 INFO:journalctl@ceph.mon.a.smithi079.stdout: No filesystems configured 2024-09-17T09:02:48.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T09:02:48.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:02:48.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:02:48.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:02:48.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-17T09:02:48.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 bash[29539]: add4676afaeed12f8a9808966d369f6c420791ee4ac18b758b05b11527f97597 2024-09-17T09:02:48.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 systemd[1]: Started Ceph mon.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:02:48.448 INFO:teuthology.orchestra.run.smithi079.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-17T09:02:48.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T09:02:48.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: monmap e1: 1 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:02:48.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: fsmap 2024-09-17T09:02:48.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: osdmap e1: 0 total, 0 up, 0 in 2024-09-17T09:02:48.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:48 smithi079 ceph-mon[29577]: mgrmap e1: no daemons active 2024-09-17T09:02:49.465 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-17T09:02:49.487 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:02:49.487 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mgr... 2024-09-17T09:02:49.487 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-17T09:02:49.487 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-17T09:02:49.736 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y 2024-09-17T09:02:49.736 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-17T09:02:49.878 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-721af936-74d3-11ef-bceb-c7b262605968.target.wants/ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-721af936-74d3-11ef-bceb-c7b262605968@.service. 2024-09-17T09:02:50.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:49 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4056604040' entity='client.admin' 2024-09-17T09:02:50.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:49 smithi079 systemd[1]: Starting Ceph mgr.y for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:02:50.392 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 podman[29813]: 2024-09-17 09:02:50.021319887 +0000 UTC m=+0.015310927 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:02:50.392 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 podman[29813]: 2024-09-17 09:02:50.122830935 +0000 UTC m=+0.116821954 container create cb364aaf426c625ee8fd9db02929fa09a36d24f80a5193f2fe2128a2fe6a289e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T09:02:50.392 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 podman[29813]: 2024-09-17 09:02:50.325365151 +0000 UTC m=+0.319356178 container init cb364aaf426c625ee8fd9db02929fa09a36d24f80a5193f2fe2128a2fe6a289e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-17T09:02:50.392 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 podman[29813]: 2024-09-17 09:02:50.330160693 +0000 UTC m=+0.324151719 container start cb364aaf426c625ee8fd9db02929fa09a36d24f80a5193f2fe2128a2fe6a289e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:02:50.393 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 bash[29813]: cb364aaf426c625ee8fd9db02929fa09a36d24f80a5193f2fe2128a2fe6a289e 2024-09-17T09:02:50.405 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T09:02:50.406 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-09-17T09:02:50.413 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T09:02:50.413 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-17T09:02:50.413 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-09-17T09:02:50.413 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-09-17T09:02:50.421 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T09:02:50.421 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-09-17T09:02:50.429 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T09:02:50.429 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-17T09:02:50.429 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-09-17T09:02:50.429 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-17T09:02:50.429 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr to start... 2024-09-17T09:02:50.429 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr... 2024-09-17T09:02:50.659 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 systemd[1]: Started Ceph mgr.y for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:02:50.659 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:50.451+0000 7f02b4286040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:02:50.659 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:50.508+0000 7f02b4286040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:02:50.659 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:50.658+0000 7f02b4286040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:02:50.968 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:50.729+0000 7f02b4286040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:02:50.968 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:50 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:50.877+0000 7f02b4286040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:02:51.264 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:51 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4260614514' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T09:02:51.264 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:51 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:51.026+0000 7f02b4286040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:02:51.578 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:51 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:51.338+0000 7f02b4286040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:02:51.579 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:51 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:51.397+0000 7f02b4286040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "721af936-74d3-11ef-bceb-c7b262605968", 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-17T09:02:51.590 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T09:02:51.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T09:02:51.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T09:02:51.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T09:02:43.786136+0000", 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-17T09:02:51.594 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (1/15)... 2024-09-17T09:02:51.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:51 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:51.578+0000 7f02b4286040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:02:52.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:52 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:52.441+0000 7f02b4286040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:02:52.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:52 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:52.505+0000 7f02b4286040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:02:53.458 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:53.207+0000 7f02b4286040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:02:53.459 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:53.320+0000 7f02b4286040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:02:53.459 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:53.458+0000 7f02b4286040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:02:53.711 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:53.517+0000 7f02b4286040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:02:53.712 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:53.672+0000 7f02b4286040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:02:54.009 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:53.738+0000 7f02b4286040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:02:54.009 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:53.799+0000 7f02b4286040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:02:54.009 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:53.960+0000 7f02b4286040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:02:54.301 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:54 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/474118591' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T09:02:54.301 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:54.026+0000 7f02b4286040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:02:54.301 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:54.095+0000 7f02b4286040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:02:54.301 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:54.182+0000 7f02b4286040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:02:54.301 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:54.242+0000 7f02b4286040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:02:54.302 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:54.300+0000 7f02b4286040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:02:54.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "721af936-74d3-11ef-bceb-c7b262605968", 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T09:02:54.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:02:54.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T09:02:54.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T09:02:54.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T09:02:54.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T09:02:54.765 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T09:02:54.765 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:54.766 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T09:02:54.767 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:02:54.767 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T09:02:54.767 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T09:02:54.767 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:54.767 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T09:02:54.767 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-17T09:02:54.767 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T09:02:54.767 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T09:02:54.767 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T09:02:43.786136+0000", 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-17T09:02:54.768 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (2/15)... 2024-09-17T09:02:55.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:02:55.077+0000 7f02b4286040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:02:55.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: Activating manager daemon y 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: mgrmap e2: y(active, starting, since 0.00274061s) 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: Manager daemon y is now available 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' 2024-09-17T09:02:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:55 smithi079 ceph-mon[29577]: from='mgr.14100 172.21.15.79:0/3755226160' entity='mgr.y' 2024-09-17T09:02:57.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:57 smithi079 ceph-mon[29577]: mgrmap e3: y(active, since 1.00707s) 2024-09-17T09:02:58.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:02:58 smithi079 ceph-mon[29577]: mgrmap e4: y(active, since 2s) 2024-09-17T09:03:00.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:00 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2017524045' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T09:03:05.547 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "721af936-74d3-11ef-bceb-c7b262605968", 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-17T09:03:05.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T09:03:05.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T09:03:05.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T09:02:43.786136+0000", 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-17T09:03:05.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T09:03:05.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-17T09:03:05.552 INFO:teuthology.orchestra.run.smithi079.stdout:mgr is available 2024-09-17T09:03:06.460 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:06 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1165231925' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-17T09:03:06.689 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:03:06.689 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-09-17T09:03:06.689 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-17T09:03:06.690 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling cephadm module... 2024-09-17T09:03:07.674 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:07 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3190255833' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-17T09:03:07.675 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:07 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ignoring --setuser ceph since I am not root 2024-09-17T09:03:07.675 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:07 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ignoring --setgroup ceph since I am not root 2024-09-17T09:03:07.675 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:07 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:07.496+0000 7f9f293a3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:03:07.675 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:07 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:07.551+0000 7f9f293a3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:03:07.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:07 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:07.699+0000 7f9f293a3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:03:07.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:07 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:07.763+0000 7f9f293a3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:03:07.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:07 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:07.907+0000 7f9f293a3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:03:08.197 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:08 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:08.051+0000 7f9f293a3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:03:08.600 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:08 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3190255833' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-17T09:03:08.600 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:08 smithi079 ceph-mon[29577]: mgrmap e5: y(active, since 12s) 2024-09-17T09:03:08.600 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:08 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:08.365+0000 7f9f293a3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:03:08.600 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:08 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:08.427+0000 7f9f293a3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:03:08.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:08 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:08.613+0000 7f9f293a3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:03:09.201 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-17T09:03:09.201 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-17T09:03:09.201 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T09:03:09.201 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-17T09:03:09.201 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-17T09:03:09.201 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-17T09:03:09.201 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-09-17T09:03:09.201 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 5... 2024-09-17T09:03:09.501 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:09 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2216363806' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T09:03:09.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:09 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:09.500+0000 7f9f293a3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:03:09.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:09 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:09.566+0000 7f9f293a3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:03:10.518 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:10 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:10.267+0000 7f9f293a3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:03:10.519 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:10 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:10.379+0000 7f9f293a3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:03:10.791 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:10 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:10.517+0000 7f9f293a3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:03:10.791 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:10 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:10.578+0000 7f9f293a3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:03:10.791 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:10 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:10.723+0000 7f9f293a3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:03:10.791 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:10 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:10.790+0000 7f9f293a3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:03:11.069 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:10 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:10.849+0000 7f9f293a3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:03:11.069 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:11.004+0000 7f9f293a3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:03:11.069 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:11.068+0000 7f9f293a3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:03:11.333 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:11.134+0000 7f9f293a3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:03:11.334 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:11.216+0000 7f9f293a3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:03:11.334 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:11.274+0000 7f9f293a3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:03:11.334 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:11.332+0000 7f9f293a3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:03:12.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:12.101+0000 7f9f293a3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:03:12.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: Active manager daemon y restarted 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: Activating manager daemon y 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: osdmap e2: 0 total, 0 up, 0 in 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: mgrmap e6: y(active, starting, since 0.00316783s) 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: Manager daemon y is now available 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:12.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:12.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:03:12.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:12 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:03:13.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:13 smithi079 ceph-mon[29577]: Found migration_current of "None". Setting to last migration. 2024-09-17T09:03:13.424 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:13 smithi079 ceph-mon[29577]: mgrmap e7: y(active, since 1.00498s) 2024-09-17T09:03:13.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-17T09:03:13.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-17T09:03:13.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-17T09:03:13.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-17T09:03:13.678 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 5 is available 2024-09-17T09:03:13.678 INFO:teuthology.orchestra.run.smithi079.stdout:Setting orchestrator backend to cephadm... 2024-09-17T09:03:14.301 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-mon[29577]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T09:03:14.301 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-mon[29577]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T09:03:14.302 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:14.302 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:14.596+0000 7f9e91043640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:09:03:14] ENGINE Error in HTTPServer.serve 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: Traceback (most recent call last): 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self._connections.run(self.expiration_interval) 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self._run(expiration_interval) 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:03:14.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:03:14.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: s = self.context.wrap_socket( 2024-09-17T09:03:14.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:03:14.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: return self.sslsocket_class._create( 2024-09-17T09:03:14.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:03:14.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self.do_handshake() 2024-09-17T09:03:14.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:03:14.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self._sslobj.do_handshake() 2024-09-17T09:03:14.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:03:14.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:15.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:15 smithi079 ceph-mon[29577]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:15.524 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:15 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:15.524 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:15 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:15.524 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:15 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:14] ENGINE Bus STARTING 2024-09-17T09:03:15.524 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:15 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:15.524 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:15 smithi079 ceph-mon[29577]: mgrmap e8: y(active, since 2s) 2024-09-17T09:03:16.064 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-17T09:03:16.064 INFO:teuthology.orchestra.run.smithi079.stdout:Generating ssh key... 2024-09-17T09:03:16.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:14] ENGINE Serving on http://172.21.15.79:8765 2024-09-17T09:03:16.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:14] ENGINE Serving on https://172.21.15.79:7150 2024-09-17T09:03:16.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:14] ENGINE Bus STARTED 2024-09-17T09:03:16.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:14] ENGINE Error in HTTPServer.serve 2024-09-17T09:03:16.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-17T09:03:16.436 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-17T09:03:16.437 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:03:16.438 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-17T09:03:16.438 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:03:17.085 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: Generating public/private rsa key pair. 2024-09-17T09:03:17.085 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: Your identification has been saved in /tmp/tmpph8c8y_a/key 2024-09-17T09:03:17.085 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: Your public key has been saved in /tmp/tmpph8c8y_a/key.pub 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: The key fingerprint is: 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: SHA256:XvIbPjObaWGxNPccYzNxkqe/xiwEkj9+ZI7FS8N7rRw ceph-721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: The key's randomart image is: 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: +---[RSA 3072]----+ 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: | . | 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: | + o| 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: | . * | 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: | o+.. B | 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: | S.+=++ * | 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: | . ++o Xo .| 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: | ..+.X E o| 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: | .=B B B.| 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: | .*=. *. | 2024-09-17T09:03:17.086 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:16 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: +----[SHA256]-----+ 2024-09-17T09:03:17.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:17 smithi079 ceph-mon[29577]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:17.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:17 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:17.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:17 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:18.507 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:18 smithi079 ceph-mon[29577]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:18.507 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:18 smithi079 ceph-mon[29577]: Generating ssh key... 2024-09-17T09:03:18.512 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA2gd7kE9orpdhCIeu2glYCHi64JnMM0OKtN/yDFlZWq2ct93YwfL4vFIEiSqaJDvulfVDB2dAunUnmk53fBv5PRShnGDJ7ZLJob8HOr+8sttrWj9md6uYxKVk99kNgluNtSOPGGF08ZxvGIHIE4Fr7F++QdigNLbDcNQ0Lj+7hWlZ/dD/NIu81cLBYcWegvfPV8VtRyD9x4M1mG0kUy3dMOvaPCg4A5769PCcSXFENr3Sie4LzVAL6MmpeQXOj2Gi7ZMAlJZ+S6MSWua6HDwVK9hGWmSmwwhG2ykQR3u1wSgmC0W2OFPQ6xGdDpDPvSVUHYnTq7Cx+VgJFozajmttd5sICyF4GzjuuPTlztr+KYOheG4NPTN64ITFLZd2ffV0+yYpTdGLnbfvre2pgYGRaDH7Fz2ssMeBOki+q4Q/Ls+u5AylNDqBVvGhiutaa1KGS8fDXwXI6aLB39Qxc5oyDl5Yjp7+x8dX8Yu004gZWk7eQ80qU6P5X1Tvm840kZU= ceph-721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:03:18.513 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-17T09:03:18.513 INFO:teuthology.orchestra.run.smithi079.stdout:Adding key to root@localhost authorized_keys... 2024-09-17T09:03:18.514 INFO:teuthology.orchestra.run.smithi079.stdout:Adding host smithi079... 2024-09-17T09:03:19.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:19 smithi079 ceph-mon[29577]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:20.563 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:20 smithi079 ceph-mon[29577]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:20.563 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:20 smithi079 ceph-mon[29577]: Deploying cephadm binary to smithi079 2024-09-17T09:03:22.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Added host 'smithi079' with addr '172.21.15.79' 2024-09-17T09:03:22.760 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying unmanaged mon service... 2024-09-17T09:03:23.588 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:23 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:23.588 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:23 smithi079 ceph-mon[29577]: Added host smithi079 2024-09-17T09:03:23.588 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:23 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:24.662 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-17T09:03:24.662 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying unmanaged mgr service... 2024-09-17T09:03:24.914 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:24 smithi079 ceph-mon[29577]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:24.915 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:24 smithi079 ceph-mon[29577]: Saving service mon spec with placement count:5 2024-09-17T09:03:24.915 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:24 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:24.915 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:24 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:24.915 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:24 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:26.422 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-17T09:03:26.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:26 smithi079 ceph-mon[29577]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:26.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:26 smithi079 ceph-mon[29577]: Saving service mgr spec with placement count:2 2024-09-17T09:03:26.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:26 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:26.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:26 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:28.219 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:27 smithi079 ceph-mon[29577]: from='mgr.14118 172.21.15.79:0/2820173279' entity='mgr.y' 2024-09-17T09:03:28.220 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:27 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/269626693' entity='client.admin' 2024-09-17T09:03:29.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:29 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2651346965' entity='client.admin' 2024-09-17T09:03:29.373 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling the dashboard module... 2024-09-17T09:03:30.357 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:30 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3196896049' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-17T09:03:30.357 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ignoring --setuser ceph since I am not root 2024-09-17T09:03:30.358 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ignoring --setgroup ceph since I am not root 2024-09-17T09:03:30.358 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:30.356+0000 7f6c4b8b9040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:03:30.611 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:30.417+0000 7f6c4b8b9040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:03:30.611 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:30.566+0000 7f6c4b8b9040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:03:30.911 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:30.633+0000 7f6c4b8b9040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:03:30.911 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:30.783+0000 7f6c4b8b9040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:03:31.213 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:30.931+0000 7f6c4b8b9040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:03:31.499 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:31 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3196896049' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-17T09:03:31.499 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:31 smithi079 ceph-mon[29577]: mgrmap e9: y(active, since 18s) 2024-09-17T09:03:31.499 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:31.251+0000 7f6c4b8b9040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:03:31.500 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:31.311+0000 7f6c4b8b9040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:03:31.500 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:31.498+0000 7f6c4b8b9040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:03:31.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-17T09:03:31.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-17T09:03:31.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T09:03:31.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-17T09:03:31.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-17T09:03:31.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-17T09:03:31.920 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-09-17T09:03:31.920 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 9... 2024-09-17T09:03:32.467 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:32 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4111281736' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T09:03:32.467 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:32.401+0000 7f6c4b8b9040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:03:32.468 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:32.466+0000 7f6c4b8b9040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:03:33.425 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:33.171+0000 7f6c4b8b9040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:03:33.426 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:33.283+0000 7f6c4b8b9040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:03:33.699 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:33.424+0000 7f6c4b8b9040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:03:33.699 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:33.485+0000 7f6c4b8b9040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:03:33.699 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:33.631+0000 7f6c4b8b9040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:03:33.699 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:33.698+0000 7f6c4b8b9040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:03:33.976 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:33.757+0000 7f6c4b8b9040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:03:33.976 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:33.911+0000 7f6c4b8b9040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:03:33.976 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:33.975+0000 7f6c4b8b9040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:03:34.238 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:34 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:34.041+0000 7f6c4b8b9040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:03:34.238 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:34 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:34.122+0000 7f6c4b8b9040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:03:34.238 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:34 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:34.180+0000 7f6c4b8b9040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:03:34.239 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:34 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:34.237+0000 7f6c4b8b9040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:03:35.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:35.044+0000 7f6c4b8b9040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:03:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: Active manager daemon y restarted 2024-09-17T09:03:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: Activating manager daemon y 2024-09-17T09:03:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: osdmap e3: 0 total, 0 up, 0 in 2024-09-17T09:03:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: mgrmap e10: y(active, starting, since 0.00345775s) 2024-09-17T09:03:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:03:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:03:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:03:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:03:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:03:35.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: Manager daemon y is now available 2024-09-17T09:03:35.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:35.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:03:35.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:03:36.611 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:36.454+0000 7f6bb754e640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:09:03:36] ENGINE Error in HTTPServer.serve 2024-09-17T09:03:36.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: Traceback (most recent call last): 2024-09-17T09:03:36.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:03:36.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self._connections.run(self.expiration_interval) 2024-09-17T09:03:36.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:03:36.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self._run(expiration_interval) 2024-09-17T09:03:36.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:03:36.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:03:36.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:03:36.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: s = self.context.wrap_socket( 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: return self.sslsocket_class._create( 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self.do_handshake() 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self._sslobj.do_handshake() 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:03:36.613 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:03:36 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:03:36.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-17T09:03:36.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-17T09:03:36.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-17T09:03:36.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-17T09:03:36.887 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 9 is available 2024-09-17T09:03:36.887 INFO:teuthology.orchestra.run.smithi079.stdout:Generating a dashboard self-signed certificate... 2024-09-17T09:03:37.320 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:37 smithi079 ceph-mon[29577]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T09:03:37.320 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:37 smithi079 ceph-mon[29577]: mgrmap e11: y(active, since 1.00581s) 2024-09-17T09:03:37.320 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:37 smithi079 ceph-mon[29577]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T09:03:37.321 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:37 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:38.174 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:38 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:36] ENGINE Bus STARTING 2024-09-17T09:03:38.174 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:38 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:36] ENGINE Serving on https://172.21.15.79:7150 2024-09-17T09:03:38.174 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:38 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:36] ENGINE Error in HTTPServer.serve 2024-09-17T09:03:38.174 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-17T09:03:38.174 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:03:38.174 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:03:38.174 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:03:38.175 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-17T09:03:38.176 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:03:38.176 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-17T09:03:38.176 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:03:38.176 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:38 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:36] ENGINE Serving on http://172.21.15.79:8765 2024-09-17T09:03:38.176 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:38 smithi079 ceph-mon[29577]: [17/Sep/2024:09:03:36] ENGINE Bus STARTED 2024-09-17T09:03:38.176 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:38 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:38.176 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:38 smithi079 ceph-mon[29577]: mgrmap e12: y(active, since 2s) 2024-09-17T09:03:38.176 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:38 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:38.176 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:38 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:38.177 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-17T09:03:38.177 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial admin user... 2024-09-17T09:03:39.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:39 smithi079 ceph-mon[29577]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:40.547 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$OuKxIX4HIih7rWD.AuzJDeiT2JEPonGLLNOZyY78hu/D98lnB7PkS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726563819, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-17T09:03:40.547 INFO:teuthology.orchestra.run.smithi079.stdout:Fetching dashboard port number... 2024-09-17T09:03:41.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:40 smithi079 ceph-mon[29577]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:41.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:40 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:41.669 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 8443 2024-09-17T09:03:41.678 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T09:03:41.678 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-09-17T09:03:41.684 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T09:03:41.685 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-17T09:03:41.685 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-09-17T09:03:41.685 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-17T09:03:41.686 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph Dashboard is now available at: 2024-09-17T09:03:41.686 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:41.686 INFO:teuthology.orchestra.run.smithi079.stdout: URL: https://smithi079.front.sepia.ceph.com:8443/ 2024-09-17T09:03:41.687 INFO:teuthology.orchestra.run.smithi079.stdout: User: admin 2024-09-17T09:03:41.687 INFO:teuthology.orchestra.run.smithi079.stdout: Password: rwv60o62qa 2024-09-17T09:03:41.687 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:41.687 INFO:teuthology.orchestra.run.smithi079.stdout:Saving cluster configuration to /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config directory 2024-09-17T09:03:41.687 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling autotune for osd_memory_target 2024-09-17T09:03:42.306 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:41 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2728851521' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-17T09:03:43.184 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:42 smithi079 ceph-mon[29577]: mgrmap e13: y(active, since 6s) 2024-09-17T09:03:44.264 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout:Or, if you are only running a single cluster on this host: 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout: ceph telemetry on 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:44.265 INFO:teuthology.orchestra.run.smithi079.stdout:For more information see: 2024-09-17T09:03:44.266 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:44.266 INFO:teuthology.orchestra.run.smithi079.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-17T09:03:44.266 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:44.266 INFO:teuthology.orchestra.run.smithi079.stdout:Bootstrap complete. 2024-09-17T09:03:44.297 INFO:tasks.cephadm:Fetching config... 2024-09-17T09:03:44.298 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:03:44.298 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-17T09:03:44.313 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-17T09:03:44.313 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:03:44.313 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-17T09:03:44.369 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-17T09:03:44.369 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:03:44.369 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-17T09:03:44.434 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-17T09:03:44.434 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:03:44.434 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-17T09:03:44.490 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-17T09:03:44.491 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA2gd7kE9orpdhCIeu2glYCHi64JnMM0OKtN/yDFlZWq2ct93YwfL4vFIEiSqaJDvulfVDB2dAunUnmk53fBv5PRShnGDJ7ZLJob8HOr+8sttrWj9md6uYxKVk99kNgluNtSOPGGF08ZxvGIHIE4Fr7F++QdigNLbDcNQ0Lj+7hWlZ/dD/NIu81cLBYcWegvfPV8VtRyD9x4M1mG0kUy3dMOvaPCg4A5769PCcSXFENr3Sie4LzVAL6MmpeQXOj2Gi7ZMAlJZ+S6MSWua6HDwVK9hGWmSmwwhG2ykQR3u1wSgmC0W2OFPQ6xGdDpDPvSVUHYnTq7Cx+VgJFozajmttd5sICyF4GzjuuPTlztr+KYOheG4NPTN64ITFLZd2ffV0+yYpTdGLnbfvre2pgYGRaDH7Fz2ssMeBOki+q4Q/Ls+u5AylNDqBVvGhiutaa1KGS8fDXwXI6aLB39Qxc5oyDl5Yjp7+x8dX8Yu004gZWk7eQ80qU6P5X1Tvm840kZU= ceph-721af936-74d3-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T09:03:44.564 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA2gd7kE9orpdhCIeu2glYCHi64JnMM0OKtN/yDFlZWq2ct93YwfL4vFIEiSqaJDvulfVDB2dAunUnmk53fBv5PRShnGDJ7ZLJob8HOr+8sttrWj9md6uYxKVk99kNgluNtSOPGGF08ZxvGIHIE4Fr7F++QdigNLbDcNQ0Lj+7hWlZ/dD/NIu81cLBYcWegvfPV8VtRyD9x4M1mG0kUy3dMOvaPCg4A5769PCcSXFENr3Sie4LzVAL6MmpeQXOj2Gi7ZMAlJZ+S6MSWua6HDwVK9hGWmSmwwhG2ykQR3u1wSgmC0W2OFPQ6xGdDpDPvSVUHYnTq7Cx+VgJFozajmttd5sICyF4GzjuuPTlztr+KYOheG4NPTN64ITFLZd2ffV0+yYpTdGLnbfvre2pgYGRaDH7Fz2ssMeBOki+q4Q/Ls+u5AylNDqBVvGhiutaa1KGS8fDXwXI6aLB39Qxc5oyDl5Yjp7+x8dX8Yu004gZWk7eQ80qU6P5X1Tvm840kZU= ceph-721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:03:44.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA2gd7kE9orpdhCIeu2glYCHi64JnMM0OKtN/yDFlZWq2ct93YwfL4vFIEiSqaJDvulfVDB2dAunUnmk53fBv5PRShnGDJ7ZLJob8HOr+8sttrWj9md6uYxKVk99kNgluNtSOPGGF08ZxvGIHIE4Fr7F++QdigNLbDcNQ0Lj+7hWlZ/dD/NIu81cLBYcWegvfPV8VtRyD9x4M1mG0kUy3dMOvaPCg4A5769PCcSXFENr3Sie4LzVAL6MmpeQXOj2Gi7ZMAlJZ+S6MSWua6HDwVK9hGWmSmwwhG2ykQR3u1wSgmC0W2OFPQ6xGdDpDPvSVUHYnTq7Cx+VgJFozajmttd5sICyF4GzjuuPTlztr+KYOheG4NPTN64ITFLZd2ffV0+yYpTdGLnbfvre2pgYGRaDH7Fz2ssMeBOki+q4Q/Ls+u5AylNDqBVvGhiutaa1KGS8fDXwXI6aLB39Qxc5oyDl5Yjp7+x8dX8Yu004gZWk7eQ80qU6P5X1Tvm840kZU= ceph-721af936-74d3-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T09:03:44.642 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA2gd7kE9orpdhCIeu2glYCHi64JnMM0OKtN/yDFlZWq2ct93YwfL4vFIEiSqaJDvulfVDB2dAunUnmk53fBv5PRShnGDJ7ZLJob8HOr+8sttrWj9md6uYxKVk99kNgluNtSOPGGF08ZxvGIHIE4Fr7F++QdigNLbDcNQ0Lj+7hWlZ/dD/NIu81cLBYcWegvfPV8VtRyD9x4M1mG0kUy3dMOvaPCg4A5769PCcSXFENr3Sie4LzVAL6MmpeQXOj2Gi7ZMAlJZ+S6MSWua6HDwVK9hGWmSmwwhG2ykQR3u1wSgmC0W2OFPQ6xGdDpDPvSVUHYnTq7Cx+VgJFozajmttd5sICyF4GzjuuPTlztr+KYOheG4NPTN64ITFLZd2ffV0+yYpTdGLnbfvre2pgYGRaDH7Fz2ssMeBOki+q4Q/Ls+u5AylNDqBVvGhiutaa1KGS8fDXwXI6aLB39Qxc5oyDl5Yjp7+x8dX8Yu004gZWk7eQ80qU6P5X1Tvm840kZU= ceph-721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:03:44.656 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-17T09:03:44.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:44 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1318733900' entity='client.admin' 2024-09-17T09:03:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:03:47.716 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-17T09:03:47.716 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-17T09:03:47.845 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:47.845 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:47.845 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:03:47.845 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:47.845 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:47.845 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:47.845 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:47 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3829118403' entity='client.admin' 2024-09-17T09:03:47.846 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:47.846 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:48.013 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:03:50.993 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-09-17T09:03:50.994 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:03:50.994 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-09-17T09:03:51.012 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:03:51.013 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:03:51.071 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-09-17T09:03:51.071 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch host add smithi173 2024-09-17T09:03:51.266 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:03:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:03:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:03:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:03:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:03:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:03:51.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:03:51.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:51.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:51.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:53.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:53 smithi079 ceph-mon[29577]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:03:54.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:54 smithi079 ceph-mon[29577]: Deploying cephadm binary to smithi173 2024-09-17T09:03:55.703 INFO:teuthology.orchestra.run.smithi079.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-09-17T09:03:56.257 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:56 smithi079 ceph-mon[29577]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:03:56.258 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:56 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:56.258 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:56 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:03:56.258 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:56 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:56.278 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-17T09:03:56.467 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:03:57.589 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:57 smithi079 ceph-mon[29577]: Added host smithi173 2024-09-17T09:03:57.589 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:57 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:57.589 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:57 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:03:58.240 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:03:58.240 INFO:teuthology.orchestra.run.smithi079.stdout:[{"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-09-17T09:03:58.587 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:58 smithi079 ceph-mon[29577]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:03:58.799 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-17T09:03:58.800 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-17T09:03:58.989 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:03:59.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:03:59 smithi079 ceph-mon[29577]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:04:00.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:00 smithi079 ceph-mon[29577]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:04:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:00.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:01.507 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-09-17T09:04:01.507 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:01 smithi079 ceph-mon[29577]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:04:01.507 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:01 smithi079 ceph-mon[29577]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:01.507 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:01 smithi079 ceph-mon[29577]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:04:01.507 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:01 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:01.507 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:01 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:01.507 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:01 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:01.507 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:01 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2635083861' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-17T09:04:01.993 INFO:tasks.cephadm:Adding mon.a on smithi079 2024-09-17T09:04:01.994 INFO:tasks.cephadm:Adding mon.c on smithi079 2024-09-17T09:04:01.994 INFO:tasks.cephadm:Adding mon.b on smithi173 2024-09-17T09:04:01.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi173:172.21.15.173=b' 2024-09-17T09:04:02.191 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:02.364 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:02 smithi079 ceph-mon[29577]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:04:02.364 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:02 smithi079 ceph-mon[29577]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:02.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:02 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2635083861' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T09:04:02.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:02 smithi079 ceph-mon[29577]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:04:03.817 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-09-17T09:04:04.483 DEBUG:teuthology.orchestra.run.smithi079:mon.c> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.c.service 2024-09-17T09:04:04.486 DEBUG:teuthology.orchestra.run.smithi173:mon.b> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.b.service 2024-09-17T09:04:04.488 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T09:04:04.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T09:04:04.510 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:04 smithi079 ceph-mon[29577]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:04.511 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:04 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:04.511 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:04 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:04.511 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:04 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:04.511 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:04 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:04.511 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:04 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:04.511 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:04 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:04.511 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:04 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:05.201 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:05.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:05 smithi079 ceph-mon[29577]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:04:05.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:05 smithi079 ceph-mon[29577]: Saving service mon spec with placement smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi173:172.21.15.173=b;count:3 2024-09-17T09:04:05.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:05 smithi079 ceph-mon[29577]: Deploying daemon mon.b on smithi173 2024-09-17T09:04:06.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:06 smithi079 ceph-mon[29577]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:06.909 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:04:06.909 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"721af936-74d3-11ef-bceb-c7b262605968","modified":"2024-09-17T09:02:40.771089Z","created":"2024-09-17T09:02:40.771089Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-17T09:04:06.909 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-09-17T09:04:07.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:07 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/4069949231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:04:08.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:08 smithi079 ceph-mon[29577]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:08.827 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T09:04:08.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T09:04:09.174 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:04:10.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:10 smithi079 ceph-mon[29577]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:10.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 podman[33363]: 2024-09-17 09:04:10.622764338 +0000 UTC m=+0.311762851 container create 6f8b6ca97f1577b52125cf31d38f11697715d2db67b175122482d05def08d7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-17T09:04:11.102 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 podman[33363]: 2024-09-17 09:04:10.8168792 +0000 UTC m=+0.505877721 container init 6f8b6ca97f1577b52125cf31d38f11697715d2db67b175122482d05def08d7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-17T09:04:11.102 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 podman[33363]: 2024-09-17 09:04:10.821598581 +0000 UTC m=+0.510597096 container start 6f8b6ca97f1577b52125cf31d38f11697715d2db67b175122482d05def08d7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T09:04:11.102 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 7 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: pidfile_write: ignore empty --pid-file 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: load: jerasure load: lrc 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: RocksDB version: 7.9.2 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Git sha 0 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: DB SUMMARY 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: DB Session ID: 0KQJ2D38V6IYWMSTKDQE 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: CURRENT file: CURRENT 2024-09-17T09:04:11.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T09:04:11.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-17T09:04:11.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-17T09:04:11.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-17T09:04:11.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.error_if_exists: 0 2024-09-17T09:04:11.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.create_if_missing: 0 2024-09-17T09:04:11.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.paranoid_checks: 1 2024-09-17T09:04:11.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T09:04:11.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T09:04:11.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T09:04:11.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.env: 0x5588cda38bc0 2024-09-17T09:04:11.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T09:04:11.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.info_log: 0x5588ce6c8be0 2024-09-17T09:04:11.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T09:04:11.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.statistics: (nil) 2024-09-17T09:04:11.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.use_fsync: 0 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_log_file_size: 0 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.allow_fallocate: 1 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.use_direct_reads: 0 2024-09-17T09:04:11.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.db_log_dir: 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.wal_dir: 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:04:11.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.write_buffer_manager: 0x5588ce6d6320 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.unordered_write: 0 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:04:11.108 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:04:11.109 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.row_cache: None 2024-09-17T09:04:11.109 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.wal_filter: None 2024-09-17T09:04:11.110 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.two_write_queues: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.wal_compression: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.atomic_flush: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.log_readahead_size: 0 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T09:04:11.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_background_jobs: 2 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_background_compactions: -1 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_subcompactions: 1 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:04:11.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_open_files: -1 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_background_flushes: -1 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Compression algorithms supported: 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: kZSTD supported: 0 2024-09-17T09:04:11.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: kXpressCompression supported: 0 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: kBZip2Compression supported: 0 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: kLZ4Compression supported: 1 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: kZlibCompression supported: 1 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: kSnappyCompression supported: 1 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.merge_operator: 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_filter: None 2024-09-17T09:04:11.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5588ce6c87e0) 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks: 1 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_top_level_index_and_filter: 1 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_type: 0 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_index_type: 0 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_shortening: 1 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: checksum: 4 2024-09-17T09:04:11.115 INFO:journalctl@ceph.mon.b.smithi173.stdout: no_block_cache: 0 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache: 0x5588ce6bd350 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_name: BinnedLRUCache 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_options: 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: capacity : 536870912 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_shard_bits : 4 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: strict_capacity_limit : 0 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: high_pri_pool_ratio: 0.000 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_compressed: (nil) 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: persistent_cache: (nil) 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size: 4096 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size_deviation: 10 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_restart_interval: 16 2024-09-17T09:04:11.116 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_block_restart_interval: 1 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: metadata_block_size: 4096 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: partition_filters: 0 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: use_delta_encoding: 1 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: filter_policy: bloomfilter 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: whole_key_filtering: 1 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: verify_compression: 0 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: read_amp_bytes_per_bit: 0 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: format_version: 5 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_index_compression: 1 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_align: 0 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: max_auto_readahead_size: 262144 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: prepopulate_block_cache: 0 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: initial_auto_readahead_size: 8192 2024-09-17T09:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression: NoCompression 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.num_levels: 7 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:04:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:04:11.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:04:11.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:04:11.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:04:11.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T09:04:11.123 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.inplace_update_support: 0 2024-09-17T09:04:11.123 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:04:11.123 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:04:11.123 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:04:11.123 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:04:11.123 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.bloom_locality: 0 2024-09-17T09:04:11.123 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.max_successive_merges: 0 2024-09-17T09:04:11.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:04:11.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:04:11.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T09:04:11.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:04:11.125 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.ttl: 2592000 2024-09-17T09:04:11.125 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:04:11.125 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T09:04:11.125 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T09:04:11.125 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.enable_blob_files: false 2024-09-17T09:04:11.126 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.min_blob_size: 0 2024-09-17T09:04:11.127 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T09:04:11.127 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T09:04:11.127 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T09:04:11.127 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T09:04:11.127 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T09:04:11.127 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T09:04:11.128 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T09:04:11.128 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T09:04:11.128 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T09:04:11.128 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-17T09:04:11.128 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 091fc84c-bf80-45e9-9281-408530f60ff8 2024-09-17T09:04:11.128 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726563850848644, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-17T09:04:11.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-17T09:04:11.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726563850849063, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726563850, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "091fc84c-bf80-45e9-9281-408530f60ff8", "db_session_id": "0KQJ2D38V6IYWMSTKDQE", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-17T09:04:11.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726563850849156, "job": 1, "event": "recovery_finished"} 2024-09-17T09:04:11.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5588ce6e8e00 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: DB pointer 0x5588ce7dc000 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** DB Stats ** 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:04:11.130 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T09:04:11.131 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-09-17T09:04:11.132 INFO:journalctl@ceph.mon.b.smithi173.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T09:04:11.132 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-09-17T09:04:11.132 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:04:11.132 INFO:journalctl@ceph.mon.b.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:04:11.132 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:04:11.132 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:04:11.132 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:04:11.132 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:04:11.132 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:04:11.133 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:04:11.133 INFO:journalctl@ceph.mon.b.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:04:11.133 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache BinnedLRUCache@0x5588ce6bd350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-17T09:04:11.133 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-17T09:04:11.133 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-09-17T09:04:11.133 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:04:11.133 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-17T09:04:11.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: using public_addr v2:172.21.15.173:0/0 -> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T09:04:11.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: starting mon.b rank -1 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:04:11.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(???) e0 preinit fsid 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:04:11.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).mds e1 new map 2024-09-17T09:04:11.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-17T09:04:11.134 INFO:journalctl@ceph.mon.b.smithi173.stdout: e1 2024-09-17T09:04:11.134 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T09:04:11.135 INFO:journalctl@ceph.mon.b.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T09:04:11.135 INFO:journalctl@ceph.mon.b.smithi173.stdout: legacy client fscid: -1 2024-09-17T09:04:11.135 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-09-17T09:04:11.135 INFO:journalctl@ceph.mon.b.smithi173.stdout: No filesystems configured 2024-09-17T09:04:11.135 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-17T09:04:11.135 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-17T09:04:11.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-17T09:04:11.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-17T09:04:11.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-17T09:04:11.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-17T09:04:11.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2635083861' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:11.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2635083861' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: Saving service mon spec with placement smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi173:172.21.15.173=b;count:3 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: Deploying daemon mon.b on smithi173 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:11.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/4069949231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:04:11.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:11.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:11.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 ceph-mon[33421]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-17T09:04:11.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 bash[33363]: 6f8b6ca97f1577b52125cf31d38f11697715d2db67b175122482d05def08d7f2 2024-09-17T09:04:11.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:10 smithi173 systemd[1]: Started Ceph mon.b for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:04:16.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:04:16.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: mon.a calling monitor election 2024-09-17T09:04:16.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:16.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: mon.b calling monitor election 2024-09-17T09:04:16.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:16.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:16.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-17T09:04:16.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: monmap e2: 2 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:04:16.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: fsmap 2024-09-17T09:04:16.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:04:16.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: mgrmap e13: y(active, since 40s) 2024-09-17T09:04:16.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: overall HEALTH_OK 2024-09-17T09:04:16.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:16.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:16.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:16.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:16.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:16.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: mon.a calling monitor election 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: mon.b calling monitor election 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: monmap e2: 2 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: fsmap 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: mgrmap e13: y(active, since 40s) 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: overall HEALTH_OK 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:17.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:16 smithi079 ceph-mon[29577]: Deploying daemon mon.c on smithi079 2024-09-17T09:04:17.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:16 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:17.220 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:16 smithi173 ceph-mon[33421]: Deploying daemon mon.c on smithi079 2024-09-17T09:04:17.220 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:16 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:17.374 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:04:17.374 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"721af936-74d3-11ef-bceb-c7b262605968","modified":"2024-09-17T09:04:10.871336Z","created":"2024-09-17T09:02:40.771089Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T09:04:17.374 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-09-17T09:04:17.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:17 smithi173 ceph-mon[33421]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:17.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:17 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/1247062274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:04:18.259 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:17 smithi079 ceph-mon[29577]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:18.259 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:17 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/1247062274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:04:18.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:18 smithi079 podman[40800]: 2024-09-17 09:04:18.720259902 +0000 UTC m=+0.015709307 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:04:19.018 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T09:04:19.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T09:04:19.235 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 podman[40800]: 2024-09-17 09:04:19.005627394 +0000 UTC m=+0.301076794 container init 7d28a3dedfcafb44ca5a3f5f45dcd718461898ef99b229b1774a80197a6c34ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 podman[40800]: 2024-09-17 09:04:19.009833713 +0000 UTC m=+0.305283115 container start 7d28a3dedfcafb44ca5a3f5f45dcd718461898ef99b229b1774a80197a6c34ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2) 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 7 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pidfile_write: ignore empty --pid-file 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: load: jerasure load: lrc 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: RocksDB version: 7.9.2 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Git sha 0 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T09:04:19.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: DB SUMMARY 2024-09-17T09:04:19.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: DB Session ID: 5ORLDABMUDVIUIXJPVGZ 2024-09-17T09:04:19.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: CURRENT file: CURRENT 2024-09-17T09:04:19.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T09:04:19.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-17T09:04:19.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-17T09:04:19.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-17T09:04:19.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.error_if_exists: 0 2024-09-17T09:04:19.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.create_if_missing: 0 2024-09-17T09:04:19.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.paranoid_checks: 1 2024-09-17T09:04:19.364 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T09:04:19.364 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T09:04:19.364 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T09:04:19.364 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.env: 0x5617bdde1bc0 2024-09-17T09:04:19.364 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T09:04:19.364 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.info_log: 0x5617bf01abe0 2024-09-17T09:04:19.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T09:04:19.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.statistics: (nil) 2024-09-17T09:04:19.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.use_fsync: 0 2024-09-17T09:04:19.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_log_file_size: 0 2024-09-17T09:04:19.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.allow_fallocate: 1 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.use_direct_reads: 0 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:04:19.367 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.db_log_dir: 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.wal_dir: 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:04:19.368 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.write_buffer_manager: 0x5617bf028320 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.unordered_write: 0 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:04:19.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.row_cache: None 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.wal_filter: None 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.two_write_queues: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.wal_compression: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.atomic_flush: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.log_readahead_size: 0 2024-09-17T09:04:19.370 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T09:04:19.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T09:04:19.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T09:04:19.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T09:04:19.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T09:04:19.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T09:04:19.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T09:04:19.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_background_jobs: 2 2024-09-17T09:04:19.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_background_compactions: -1 2024-09-17T09:04:19.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_subcompactions: 1 2024-09-17T09:04:19.372 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:04:19.372 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:04:19.372 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_open_files: -1 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_background_flushes: -1 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Compression algorithms supported: 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: kZSTD supported: 0 2024-09-17T09:04:19.373 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: kXpressCompression supported: 0 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: kBZip2Compression supported: 0 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: kLZ4Compression supported: 1 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: kZlibCompression supported: 1 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: kSnappyCompression supported: 1 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.merge_operator: 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_filter: None 2024-09-17T09:04:19.374 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617bf01a7e0) 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: cache_index_and_filter_blocks: 1 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: pin_top_level_index_and_filter: 1 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: index_type: 0 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: data_block_index_type: 0 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: index_shortening: 1 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: checksum: 4 2024-09-17T09:04:19.375 INFO:journalctl@ceph.mon.c.smithi079.stdout: no_block_cache: 0 2024-09-17T09:04:19.376 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache: 0x5617bf00f350 2024-09-17T09:04:19.376 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache_name: BinnedLRUCache 2024-09-17T09:04:19.376 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache_options: 2024-09-17T09:04:19.376 INFO:journalctl@ceph.mon.c.smithi079.stdout: capacity : 536870912 2024-09-17T09:04:19.376 INFO:journalctl@ceph.mon.c.smithi079.stdout: num_shard_bits : 4 2024-09-17T09:04:19.376 INFO:journalctl@ceph.mon.c.smithi079.stdout: strict_capacity_limit : 0 2024-09-17T09:04:19.376 INFO:journalctl@ceph.mon.c.smithi079.stdout: high_pri_pool_ratio: 0.000 2024-09-17T09:04:19.376 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache_compressed: (nil) 2024-09-17T09:04:19.376 INFO:journalctl@ceph.mon.c.smithi079.stdout: persistent_cache: (nil) 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_size: 4096 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_size_deviation: 10 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_restart_interval: 16 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: index_block_restart_interval: 1 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: metadata_block_size: 4096 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: partition_filters: 0 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: use_delta_encoding: 1 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: filter_policy: bloomfilter 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: whole_key_filtering: 1 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: verify_compression: 0 2024-09-17T09:04:19.377 INFO:journalctl@ceph.mon.c.smithi079.stdout: read_amp_bytes_per_bit: 0 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout: format_version: 5 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout: enable_index_compression: 1 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_align: 0 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout: max_auto_readahead_size: 262144 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout: prepopulate_block_cache: 0 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout: initial_auto_readahead_size: 8192 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression: NoCompression 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:04:19.378 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.num_levels: 7 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:04:19.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:04:19.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T09:04:19.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:04:19.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.inplace_update_support: 0 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.bloom_locality: 0 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.max_successive_merges: 0 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.ttl: 2592000 2024-09-17T09:04:19.383 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.enable_blob_files: false 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.min_blob_size: 0 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T09:04:19.384 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5c55efe6-ab43-492b-afec-67516198f7bc 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726563859036668, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726563859037052, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726563859, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5c55efe6-ab43-492b-afec-67516198f7bc", "db_session_id": "5ORLDABMUDVIUIXJPVGZ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726563859037144, "job": 1, "event": "recovery_finished"} 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5617bf03ae00 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: DB pointer 0x5617bf12e000 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T09:04:19.385 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** DB Stats ** 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** Compaction Stats [default] ** 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T09:04:19.386 INFO:journalctl@ceph.mon.c.smithi079.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** Compaction Stats [default] ** 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:04:19.387 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout: Block cache BinnedLRUCache@0x5617bf00f350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: using public_addrv [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: starting mon.c rank -1 at public addrs [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] at bind addrs [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(???) e0 preinit fsid 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:04:19.388 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).mds e1 new map 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout: e1 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout: legacy client fscid: -1 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout: No filesystems configured 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-17T09:04:19.389 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:19.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2635083861' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2635083861' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:19.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: Saving service mon spec with placement smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi173:172.21.15.173=b;count:3 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: Deploying daemon mon.b on smithi173 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/4069949231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.a calling monitor election 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:19.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.b calling monitor election 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: monmap e2: 2 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: fsmap 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mgrmap e13: y(active, since 40s) 2024-09-17T09:04:19.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: overall HEALTH_OK 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: Deploying daemon mon.c on smithi079 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/1247062274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 ceph-mon[40839]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 bash[40800]: 7d28a3dedfcafb44ca5a3f5f45dcd718461898ef99b229b1774a80197a6c34ac 2024-09-17T09:04:19.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:19 smithi079 systemd[1]: Started Ceph mon.c for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:04:24.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:24.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:04:24.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: mon.a calling monitor election 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: mon.b calling monitor election 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: mon.c calling monitor election 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T09:04:24.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:04:24.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: fsmap 2024-09-17T09:04:24.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:04:24.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: mgrmap e13: y(active, since 49s) 2024-09-17T09:04:24.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: overall HEALTH_OK 2024-09-17T09:04:24.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:04:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: mon.a calling monitor election 2024-09-17T09:04:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: mon.b calling monitor election 2024-09-17T09:04:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: mon.c calling monitor election 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: fsmap 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: mgrmap e13: y(active, since 49s) 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: overall HEALTH_OK 2024-09-17T09:04:24.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: mon.a calling monitor election 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: mon.b calling monitor election 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: mon.c calling monitor election 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: fsmap 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: mgrmap e13: y(active, since 49s) 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: overall HEALTH_OK 2024-09-17T09:04:24.615 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.616 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.616 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.616 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:24.617 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T09:04:24.618 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"721af936-74d3-11ef-bceb-c7b262605968","modified":"2024-09-17T09:04:19.059184Z","created":"2024-09-17T09:02:40.771089Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-17T09:04:24.618 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-09-17T09:04:25.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:25 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:25.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:25 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/1786321632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:04:25.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:25 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:25.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:25 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:25.300 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-17T09:04:25.300 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-17T09:04:25.491 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:04:25.601 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:25 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:25.601 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:25 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/1786321632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:04:25.601 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:25 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:25.601 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:25 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:25.602 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:25 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:25.602 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:25 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/1786321632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:04:25.602 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:25 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:04:25.602 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:25 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.567 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:26.567 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.568 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.568 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.568 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.568 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:26.569 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:26.569 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.569 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.569 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.569 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.570 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:26.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.571 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.571 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.571 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:26.571 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:26.571 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.572 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.572 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.572 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:26.572 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:04:27.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:04:27.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:27.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:27.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T09:04:27.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:27.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:04:27.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:27.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:27 smithi173 ceph-mon[33421]: Reconfiguring daemon mon.a on smithi079 2024-09-17T09:04:27.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:04:27.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:04:27.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:27.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:27.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[29577]: Reconfiguring daemon mon.a on smithi079 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.542 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.542 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:27.542 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T09:04:27.542 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:27.542 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:04:27.542 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:27.542 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:27 smithi079 ceph-mon[40839]: Reconfiguring daemon mon.a on smithi079 2024-09-17T09:04:27.797 INFO:teuthology.orchestra.run.smithi079.stdout:# minimal ceph.conf for 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:04:27.798 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-09-17T09:04:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:04:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: mon_host = [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-09-17T09:04:28.462 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[29577]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:28.462 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3801959063' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[40839]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3801959063' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:28.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:04:28.464 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:28 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:28.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:28 smithi173 ceph-mon[33421]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:28.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:28 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3801959063' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:28.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:28 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:28.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:28 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:28.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:28 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:28.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:28 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:04:28.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:28 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:28.700 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-17T09:04:28.700 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:04:28.700 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T09:04:28.728 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:04:28.728 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:04:28.793 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:04:28.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T09:04:28.827 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:04:28.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:04:28.897 INFO:tasks.cephadm:Adding mgr.y on smithi079 2024-09-17T09:04:28.897 INFO:tasks.cephadm:Adding mgr.x on smithi173 2024-09-17T09:04:28.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi079=y;smithi173=x' 2024-09-17T09:04:29.126 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:04:29.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:29 smithi079 ceph-mon[29577]: Reconfiguring mon.c (monmap changed)... 2024-09-17T09:04:29.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:29 smithi079 ceph-mon[29577]: Reconfiguring daemon mon.c on smithi079 2024-09-17T09:04:29.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:29 smithi079 ceph-mon[40839]: Reconfiguring mon.c (monmap changed)... 2024-09-17T09:04:29.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:29 smithi079 ceph-mon[40839]: Reconfiguring daemon mon.c on smithi079 2024-09-17T09:04:29.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:29 smithi173 ceph-mon[33421]: Reconfiguring mon.c (monmap changed)... 2024-09-17T09:04:29.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:29 smithi173 ceph-mon[33421]: Reconfiguring daemon mon.c on smithi079 2024-09-17T09:04:30.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:30 smithi173 ceph-mon[33421]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:30.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:30 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:30.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:30 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:30.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:30 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:30.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:30 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:04:30.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:30 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:30.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[29577]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:30.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:30.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:30.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:30.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:04:30.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:30.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[40839]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:30.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:30.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:30.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:04:30.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:04:30.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:30 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:30.853 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-09-17T09:04:31.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:31 smithi173 ceph-mon[33421]: Reconfiguring mon.b (monmap changed)... 2024-09-17T09:04:31.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:31 smithi173 ceph-mon[33421]: Reconfiguring daemon mon.b on smithi173 2024-09-17T09:04:31.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:31 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:31.558 DEBUG:teuthology.orchestra.run.smithi173:mgr.x> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.x.service 2024-09-17T09:04:31.560 INFO:tasks.cephadm:Deploying OSDs... 2024-09-17T09:04:31.560 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:04:31.560 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T09:04:31.580 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T09:04:31.580 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-09-17T09:04:31.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:31 smithi079 ceph-mon[40839]: Reconfiguring mon.b (monmap changed)... 2024-09-17T09:04:31.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:31 smithi079 ceph-mon[40839]: Reconfiguring daemon mon.b on smithi173 2024-09-17T09:04:31.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:31 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:31.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:31 smithi079 ceph-mon[29577]: Reconfiguring mon.b (monmap changed)... 2024-09-17T09:04:31.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:31 smithi079 ceph-mon[29577]: Reconfiguring daemon mon.b on smithi173 2024-09-17T09:04:31.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:31 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:31.629 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T09:04:31.630 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:04:31.630 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 960 Links: 1 2024-09-17T09:04:31.630 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:04:31.630 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:04:31.630 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-17 09:04:18.546586796 +0000 2024-09-17T09:04:31.630 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-17 08:59:10.965987296 +0000 2024-09-17T09:04:31.630 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-17 08:59:10.965987296 +0000 2024-09-17T09:04:31.630 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-09-17 08:59:10.965987296 +0000 2024-09-17T09:04:31.631 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T09:04:31.697 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-17T09:04:31.697 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-17T09:04:31.697 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000297505 s, 1.7 MB/s 2024-09-17T09:04:31.699 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T09:04:31.757 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-09-17T09:04:31.814 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T09:04:31.814 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:04:31.815 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 965 Links: 1 2024-09-17T09:04:31.815 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:04:31.815 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:04:31.815 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-17 09:04:18.546586796 +0000 2024-09-17T09:04:31.815 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-17 08:59:10.965987296 +0000 2024-09-17T09:04:31.815 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-17 08:59:10.965987296 +0000 2024-09-17T09:04:31.815 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-09-17 08:59:10.965987296 +0000 2024-09-17T09:04:31.815 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T09:04:31.880 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-17T09:04:31.880 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-17T09:04:31.880 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000206122 s, 2.5 MB/s 2024-09-17T09:04:31.881 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T09:04:31.939 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-09-17T09:04:31.995 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T09:04:31.996 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:04:31.996 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 973 Links: 1 2024-09-17T09:04:31.996 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:04:31.996 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:04:31.996 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-17 09:04:18.547586772 +0000 2024-09-17T09:04:31.996 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-17 08:59:10.966987152 +0000 2024-09-17T09:04:31.996 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-17 08:59:10.966987152 +0000 2024-09-17T09:04:31.996 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-09-17 08:59:10.966987152 +0000 2024-09-17T09:04:31.996 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T09:04:32.060 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-17T09:04:32.060 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-17T09:04:32.060 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000224802 s, 2.3 MB/s 2024-09-17T09:04:32.061 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T09:04:32.120 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-09-17T09:04:32.175 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T09:04:32.175 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:04:32.175 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-17T09:04:32.175 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:04:32.175 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:04:32.176 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-17 09:04:18.547586772 +0000 2024-09-17T09:04:32.176 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-17 08:59:10.966987152 +0000 2024-09-17T09:04:32.176 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-17 08:59:10.966987152 +0000 2024-09-17T09:04:32.176 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-09-17 08:59:10.966987152 +0000 2024-09-17T09:04:32.176 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T09:04:32.239 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-17T09:04:32.239 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-17T09:04:32.239 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.00020501 s, 2.5 MB/s 2024-09-17T09:04:32.240 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T09:04:32.297 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:04:32.297 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T09:04:32.315 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T09:04:32.315 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-09-17T09:04:32.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:32 smithi079 ceph-mon[29577]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi079=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:04:32.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:32 smithi079 ceph-mon[29577]: Saving service mgr spec with placement smithi079=y;smithi173=x;count:2 2024-09-17T09:04:32.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:32 smithi079 ceph-mon[29577]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:32.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:32 smithi079 ceph-mon[40839]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi079=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:04:32.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:32 smithi079 ceph-mon[40839]: Saving service mgr spec with placement smithi079=y;smithi173=x;count:2 2024-09-17T09:04:32.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:32 smithi079 ceph-mon[40839]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T09:04:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:04:32.372 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 977 Links: 1 2024-09-17T09:04:32.372 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:04:32.372 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:04:32.372 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 09:04:09.802682178 +0000 2024-09-17T09:04:32.372 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 08:59:11.282402348 +0000 2024-09-17T09:04:32.373 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 08:59:11.282402348 +0000 2024-09-17T09:04:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 08:59:11.282402348 +0000 2024-09-17T09:04:32.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T09:04:32.434 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:32 smithi173 ceph-mon[33421]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi079=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:04:32.434 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:32 smithi173 ceph-mon[33421]: Saving service mgr spec with placement smithi079=y;smithi173=x;count:2 2024-09-17T09:04:32.434 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:32 smithi173 ceph-mon[33421]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:32.436 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T09:04:32.437 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T09:04:32.437 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000187236 s, 2.7 MB/s 2024-09-17T09:04:32.438 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T09:04:32.495 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-09-17T09:04:32.551 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T09:04:32.551 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:04:32.551 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-17T09:04:32.551 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:04:32.551 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:04:32.552 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 09:04:09.803682155 +0000 2024-09-17T09:04:32.552 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 08:59:11.281402371 +0000 2024-09-17T09:04:32.552 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 08:59:11.281402371 +0000 2024-09-17T09:04:32.552 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 08:59:11.281402371 +0000 2024-09-17T09:04:32.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T09:04:32.616 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T09:04:32.616 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T09:04:32.616 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000185475 s, 2.8 MB/s 2024-09-17T09:04:32.617 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T09:04:32.673 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-09-17T09:04:32.728 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T09:04:32.729 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:04:32.729 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 983 Links: 1 2024-09-17T09:04:32.729 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:04:32.729 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:04:32.729 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 09:04:09.803682155 +0000 2024-09-17T09:04:32.729 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 08:59:11.282402348 +0000 2024-09-17T09:04:32.729 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 08:59:11.282402348 +0000 2024-09-17T09:04:32.729 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 08:59:11.282402348 +0000 2024-09-17T09:04:32.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T09:04:32.791 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T09:04:32.792 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T09:04:32.792 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000223238 s, 2.3 MB/s 2024-09-17T09:04:32.794 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T09:04:32.850 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-09-17T09:04:32.906 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T09:04:32.906 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:04:32.906 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 972 Links: 1 2024-09-17T09:04:32.906 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:04:32.906 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:04:32.906 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 09:04:09.803682155 +0000 2024-09-17T09:04:32.906 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 08:59:11.282402348 +0000 2024-09-17T09:04:32.906 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 08:59:11.282402348 +0000 2024-09-17T09:04:32.906 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 08:59:11.282402348 +0000 2024-09-17T09:04:32.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T09:04:32.969 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T09:04:32.969 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T09:04:32.969 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000218514 s, 2.3 MB/s 2024-09-17T09:04:32.970 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T09:04:33.027 INFO:tasks.cephadm:Deploying osd.0 on smithi079 with /dev/vg_nvme/lv_4... 2024-09-17T09:04:33.027 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-17T09:04:33.225 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:04:33.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:33.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:33.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:33.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:33.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:33.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:33.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:04:33.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T09:04:33.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:04:33.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: Deploying daemon mgr.x on smithi173 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[29577]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T09:04:33.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:04:33.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:33.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: Deploying daemon mgr.x on smithi173 2024-09-17T09:04:33.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:33 smithi079 ceph-mon[40839]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:33.723 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:33.723 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:33.723 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:33.723 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:33.724 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:33.724 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:33.724 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:04:33.724 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T09:04:33.724 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:04:33.724 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:33.724 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: Deploying daemon mgr.x on smithi173 2024-09-17T09:04:33.724 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:33 smithi173 ceph-mon[33421]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:34.256 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 systemd[1]: Starting Ceph mgr.x for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:04:34.663 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 podman[35300]: 2024-09-17 09:04:34.343509104 +0000 UTC m=+0.094407202 container create fa80598b963d6dd2760903eed4d79c8d838e3e6d735c90cda278dc588950226c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:04:34.663 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 podman[35300]: 2024-09-17 09:04:34.264637711 +0000 UTC m=+0.015535833 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:04:34.663 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 podman[35300]: 2024-09-17 09:04:34.554232388 +0000 UTC m=+0.305130494 container init fa80598b963d6dd2760903eed4d79c8d838e3e6d735c90cda278dc588950226c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_CLEAN=True) 2024-09-17T09:04:34.663 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 podman[35300]: 2024-09-17 09:04:34.559099623 +0000 UTC m=+0.309997725 container start fa80598b963d6dd2760903eed4d79c8d838e3e6d735c90cda278dc588950226c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b) 2024-09-17T09:04:34.663 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 ceph-mgr[35337]: -- 172.21.15.173:0/3788460347 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a965decd00 con 0x55a965dc1400 2024-09-17T09:04:34.663 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 bash[35300]: fa80598b963d6dd2760903eed4d79c8d838e3e6d735c90cda278dc588950226c 2024-09-17T09:04:34.925 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-17T09:04:34.965 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 systemd[1]: Started Ceph mgr.x for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:04:34.965 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:34.678+0000 7f734b63f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:04:34.965 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:34.739+0000 7f734b63f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:04:34.965 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:34.895+0000 7f734b63f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:04:34.965 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:34 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:34.964+0000 7f734b63f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:04:35.004 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-17T09:04:35.022 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-09-17T09:04:35.022 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-09-17T09:04:35.022 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0165367 s, 634 MB/s 2024-09-17T09:04:35.022 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-09-17T09:04:35.256 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:35.146+0000 7f734b63f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:04:35.582 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:35.290+0000 7f734b63f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:04:35.627 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_4 2024-09-17T09:04:35.832 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:04:35.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:35.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[40839]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:35.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:35.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[29577]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:35.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:35 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.866 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:35.612+0000 7f734b63f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:04:35.866 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:35.672+0000 7f734b63f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:04:35.866 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.866 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.866 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.866 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:35.866 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:35.866 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-mon[33421]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:35.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:36.256 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:35.864+0000 7f734b63f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:04:37.006 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:36.728+0000 7f734b63f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:36.792+0000 7f734b63f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:04:37.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:37.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:36 smithi173 ceph-mon[33421]: Reconfiguring daemon mgr.y on smithi079 2024-09-17T09:04:37.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:37.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:37.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[29577]: Reconfiguring daemon mgr.y on smithi079 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:37.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:37.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:37.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:37.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T09:04:37.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:04:37.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:04:37.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:37.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:36 smithi079 ceph-mon[40839]: Reconfiguring daemon mgr.y on smithi079 2024-09-17T09:04:37.752 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:37 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:37.500+0000 7f734b63f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:04:37.753 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:37 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:37.613+0000 7f734b63f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:04:37.753 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:37 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:37.751+0000 7f734b63f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:04:38.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:37 smithi173 ceph-mon[33421]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:38.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:37 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:38.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:37 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:37 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:38.007 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:37 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:37.813+0000 7f734b63f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:04:38.007 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:37 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:37.960+0000 7f734b63f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:04:38.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:37 smithi079 ceph-mon[29577]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:38.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:37 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:38.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:37 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:38.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:37 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:38.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:37 smithi079 ceph-mon[40839]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:38.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:37 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:38.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:37 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:38.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:37 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:38.310 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:38.028+0000 7f734b63f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:04:38.310 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:38.086+0000 7f734b63f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:04:38.310 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:38.245+0000 7f734b63f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:04:38.310 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:38.309+0000 7f734b63f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:04:38.571 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:38.374+0000 7f734b63f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:04:38.571 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:38.455+0000 7f734b63f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:04:38.571 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:38.512+0000 7f734b63f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:04:38.572 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:38.570+0000 7f734b63f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:04:38.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:38.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:38.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:38.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:04:38.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:04:38.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:38.991 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:38.991 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:38.991 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:38.991 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:04:38.991 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:04:38.991 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:38 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:39.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:39.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:39.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:39.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:04:39.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:04:39.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:38 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:39.756 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:04:39 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:04:39.340+0000 7f734b63f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:04:40.063 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[29577]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:04:40.063 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[29577]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:40.063 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[29577]: Standby manager daemon x started 2024-09-17T09:04:40.063 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:04:40.063 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:04:40.063 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:04:40.063 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:04:40.064 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[40839]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:04:40.064 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[40839]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:40.064 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[40839]: Standby manager daemon x started 2024-09-17T09:04:40.064 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:04:40.064 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:04:40.064 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:04:40.064 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:39 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:04:40.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:39 smithi173 ceph-mon[33421]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:04:40.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:39 smithi173 ceph-mon[33421]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:40.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:39 smithi173 ceph-mon[33421]: Standby manager daemon x started 2024-09-17T09:04:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:39 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:04:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:39 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:04:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:39 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:04:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:39 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.173:0/3701254943' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:04:41.078 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[29577]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-17T09:04:41.078 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:04:41.078 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3260668160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bcae75-6877-4f44-ac7f-55d7393f6fa0"}]: dispatch 2024-09-17T09:04:41.078 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bcae75-6877-4f44-ac7f-55d7393f6fa0"}]: dispatch 2024-09-17T09:04:41.078 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7bcae75-6877-4f44-ac7f-55d7393f6fa0"}]': finished 2024-09-17T09:04:41.078 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[29577]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T09:04:41.078 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:41.079 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[40839]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-17T09:04:41.079 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:04:41.079 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3260668160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bcae75-6877-4f44-ac7f-55d7393f6fa0"}]: dispatch 2024-09-17T09:04:41.079 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bcae75-6877-4f44-ac7f-55d7393f6fa0"}]: dispatch 2024-09-17T09:04:41.079 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7bcae75-6877-4f44-ac7f-55d7393f6fa0"}]': finished 2024-09-17T09:04:41.079 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[40839]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T09:04:41.079 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:40 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:41.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:40 smithi173 ceph-mon[33421]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-17T09:04:41.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:40 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:04:41.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:40 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3260668160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bcae75-6877-4f44-ac7f-55d7393f6fa0"}]: dispatch 2024-09-17T09:04:41.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:40 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bcae75-6877-4f44-ac7f-55d7393f6fa0"}]: dispatch 2024-09-17T09:04:41.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:40 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7bcae75-6877-4f44-ac7f-55d7393f6fa0"}]': finished 2024-09-17T09:04:41.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:40 smithi173 ceph-mon[33421]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T09:04:41.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:40 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:42.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:41 smithi173 ceph-mon[33421]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:42.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:41 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1610801936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:04:42.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:41 smithi079 ceph-mon[29577]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:42.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:41 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1610801936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:04:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:41 smithi079 ceph-mon[40839]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:41 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1610801936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:04:44.379 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:44 smithi079 ceph-mon[29577]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:44.379 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:44 smithi079 ceph-mon[40839]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:44.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:44 smithi173 ceph-mon[33421]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:46.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:46 smithi079 ceph-mon[29577]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:46.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:46 smithi079 ceph-mon[40839]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:46.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:46 smithi173 ceph-mon[33421]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:47.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:04:47.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:47.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:47 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:04:47.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:47 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:47.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:47 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:04:47.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:47 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:48.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:48 smithi079 ceph-mon[29577]: Deploying daemon osd.0 on smithi079 2024-09-17T09:04:48.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:48 smithi079 ceph-mon[29577]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:48.155 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:48 smithi079 ceph-mon[40839]: Deploying daemon osd.0 on smithi079 2024-09-17T09:04:48.156 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:48 smithi079 ceph-mon[40839]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:48.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:48 smithi173 ceph-mon[33421]: Deploying daemon osd.0 on smithi079 2024-09-17T09:04:48.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:48 smithi173 ceph-mon[33421]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:50.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:50 smithi079 ceph-mon[29577]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:50.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:50 smithi079 ceph-mon[40839]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:50.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:50 smithi173 ceph-mon[33421]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:52.267 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:52 smithi079 ceph-mon[40839]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:52.267 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:52 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:52.267 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:52 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:52.267 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:52 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:52.267 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:52 smithi079 ceph-mon[29577]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:52.268 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:52 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:52.268 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:52 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:52.268 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:52 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:52.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:52 smithi173 ceph-mon[33421]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:52.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:52 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:52.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:52 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:52.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:52 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:54.233 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 0 on host 'smithi079' 2024-09-17T09:04:54.245 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:54 smithi079 ceph-mon[29577]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:54.246 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:54 smithi079 ceph-mon[40839]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:54.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:54 smithi173 ceph-mon[33421]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:54.786 DEBUG:teuthology.orchestra.run.smithi079:osd.0> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.0.service 2024-09-17T09:04:54.788 INFO:tasks.cephadm:Deploying osd.1 on smithi079 with /dev/vg_nvme/lv_3... 2024-09-17T09:04:54.788 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-17T09:04:55.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[29577]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:04:55.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:55 smithi079 ceph-mon[40839]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:04:55.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:55.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:55 smithi173 ceph-mon[33421]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:04:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:04:56.584 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[29577]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:56.584 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[29577]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T09:04:56.585 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[29577]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T09:04:56.585 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:56.585 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[29577]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:04:56.585 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[40839]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:56.585 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[40839]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T09:04:56.585 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[40839]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T09:04:56.585 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:56.585 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-mon[40839]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:04:56.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:56 smithi173 ceph-mon[33421]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:56.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:56 smithi173 ceph-mon[33421]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T09:04:56.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:56 smithi173 ceph-mon[33421]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T09:04:56.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:56 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:56.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:56 smithi173 ceph-mon[33421]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:04:56.851 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0[45195]: 2024-09-17T09:04:56.740+0000 7fa50bf94640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T09:04:56.851 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:04:56 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0[45195]: 2024-09-17T09:04:56.743+0000 7fa50359e640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:04:57.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[29577]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:04:57.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[29577]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T09:04:57.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:57.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:57.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[29577]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' 2024-09-17T09:04:57.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[29577]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:57.407 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[40839]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:04:57.407 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[40839]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T09:04:57.408 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:57.408 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:57.408 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[40839]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' 2024-09-17T09:04:57.408 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:57 smithi079 ceph-mon[40839]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:57.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:57 smithi173 ceph-mon[33421]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:04:57.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:57 smithi173 ceph-mon[33421]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T09:04:57.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:57 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:57.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:57 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:57.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:57 smithi173 ceph-mon[33421]: from='osd.0 v2:172.21.15.79:6801/1619782938' entity='osd.0' 2024-09-17T09:04:57.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:57 smithi173 ceph-mon[33421]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:04:57.951 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-17T09:04:58.034 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-17T09:04:58.049 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-09-17T09:04:58.049 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-09-17T09:04:58.049 INFO:teuthology.orchestra.run.smithi079.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125696 s, 834 MB/s 2024-09-17T09:04:58.049 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-09-17T09:04:58.569 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[29577]: purged_snaps scrub starts 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[29577]: purged_snaps scrub ok 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[29577]: osd.0 v2:172.21.15.79:6801/1619782938 boot 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[29577]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[40839]: purged_snaps scrub starts 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[40839]: purged_snaps scrub ok 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[40839]: osd.0 v2:172.21.15.79:6801/1619782938 boot 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[40839]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T09:04:58.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:58 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:58.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:58 smithi173 ceph-mon[33421]: purged_snaps scrub starts 2024-09-17T09:04:58.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:58 smithi173 ceph-mon[33421]: purged_snaps scrub ok 2024-09-17T09:04:58.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:58 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:58.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:58 smithi173 ceph-mon[33421]: osd.0 v2:172.21.15.79:6801/1619782938 boot 2024-09-17T09:04:58.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:58 smithi173 ceph-mon[33421]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T09:04:58.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:58 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:04:59.111 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_3 2024-09-17T09:04:59.305 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:04:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: Detected new or changed devices on smithi079 2024-09-17T09:04:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:04:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi079 to 16064M 2024-09-17T09:04:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[40839]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: Detected new or changed devices on smithi079 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:04:59.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi079 to 16064M 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:04:59.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:04:59 smithi079 ceph-mon[29577]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: Detected new or changed devices on smithi079 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi079 to 16064M 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:00.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:00.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:00.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:00.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:04:59 smithi173 ceph-mon[33421]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T09:05:01.350 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:01 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:05:01.351 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:01 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:05:01.351 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:01 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:01.351 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:01 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:05:01.351 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:01 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:05:01.351 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:01 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:01.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:01 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:05:01.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:01 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:05:01.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:01 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:02.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:02 smithi079 ceph-mon[29577]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:05:02.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:02 smithi079 ceph-mon[29577]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:02.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:02 smithi079 ceph-mon[40839]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:05:02.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:02 smithi079 ceph-mon[40839]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:02.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:02 smithi173 ceph-mon[33421]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:05:02.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:02 smithi173 ceph-mon[33421]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:04.374 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[29577]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:04.374 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/132113855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0cf41b3-7003-426b-af5f-dcf5d7ee48d5"}]: dispatch 2024-09-17T09:05:04.374 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0cf41b3-7003-426b-af5f-dcf5d7ee48d5"}]: dispatch 2024-09-17T09:05:04.374 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0cf41b3-7003-426b-af5f-dcf5d7ee48d5"}]': finished 2024-09-17T09:05:04.374 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[29577]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T09:05:04.374 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:04.375 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3078138235' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:05:04.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[40839]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:04.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/132113855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0cf41b3-7003-426b-af5f-dcf5d7ee48d5"}]: dispatch 2024-09-17T09:05:04.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0cf41b3-7003-426b-af5f-dcf5d7ee48d5"}]: dispatch 2024-09-17T09:05:04.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0cf41b3-7003-426b-af5f-dcf5d7ee48d5"}]': finished 2024-09-17T09:05:04.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[40839]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T09:05:04.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:04.375 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:04 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3078138235' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:05:04.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:04 smithi173 ceph-mon[33421]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:04.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:04 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/132113855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0cf41b3-7003-426b-af5f-dcf5d7ee48d5"}]: dispatch 2024-09-17T09:05:04.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:04 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0cf41b3-7003-426b-af5f-dcf5d7ee48d5"}]: dispatch 2024-09-17T09:05:04.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:04 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0cf41b3-7003-426b-af5f-dcf5d7ee48d5"}]': finished 2024-09-17T09:05:04.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:04 smithi173 ceph-mon[33421]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T09:05:04.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:04 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:04.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:04 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3078138235' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:05:06.323 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:06 smithi079 ceph-mon[29577]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:06.323 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:06 smithi079 ceph-mon[40839]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:06.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:06 smithi173 ceph-mon[33421]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:08.165 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:08 smithi079 ceph-mon[29577]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:08.165 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:08 smithi079 ceph-mon[40839]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:08.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:08 smithi173 ceph-mon[33421]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:09.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:09 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:05:09.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:09 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:09.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:09 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:05:09.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:09 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:09.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:09 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:05:09.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:09 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:10.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:10 smithi173 ceph-mon[33421]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:10.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:10 smithi173 ceph-mon[33421]: Deploying daemon osd.1 on smithi079 2024-09-17T09:05:10.536 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:10 smithi079 ceph-mon[29577]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:10.536 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:10 smithi079 ceph-mon[29577]: Deploying daemon osd.1 on smithi079 2024-09-17T09:05:10.536 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:10 smithi079 ceph-mon[40839]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:10.537 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:10 smithi079 ceph-mon[40839]: Deploying daemon osd.1 on smithi079 2024-09-17T09:05:12.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:12 smithi079 ceph-mon[29577]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:12.428 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:12 smithi079 ceph-mon[40839]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:12.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:12 smithi173 ceph-mon[33421]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:14.370 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:14 smithi079 ceph-mon[29577]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:14.371 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:14 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:14.371 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:14 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:14.371 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:14 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:14.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:14 smithi079 ceph-mon[40839]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:14.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:14 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:14.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:14 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:14.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:14 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:14.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:14 smithi173 ceph-mon[33421]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:14.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:14 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:14.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:14 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:14.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:14 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:16.173 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:16 smithi079 ceph-mon[29577]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:16.173 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:16 smithi079 ceph-mon[40839]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:16.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:16 smithi173 ceph-mon[33421]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:17.065 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 1 on host 'smithi079' 2024-09-17T09:05:17.605 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.605 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.605 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='osd.1 v2:172.21.15.79:6805/4182839712' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[40839]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.606 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:17.607 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:17.607 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.607 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='osd.1 v2:172.21.15.79:6805/4182839712' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:05:17.607 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:05:17.607 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:17.607 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.607 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.607 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:17 smithi079 ceph-mon[29577]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:17.625 DEBUG:teuthology.orchestra.run.smithi079:osd.1> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.1.service 2024-09-17T09:05:17.628 INFO:tasks.cephadm:Deploying osd.2 on smithi079 with /dev/vg_nvme/lv_2... 2024-09-17T09:05:17.628 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='osd.1 v2:172.21.15.79:6805/4182839712' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:17.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:17 smithi173 ceph-mon[33421]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:18.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[40839]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T09:05:18.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[40839]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T09:05:18.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:18.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[40839]: from='osd.1 v2:172.21.15.79:6805/4182839712' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:05:18.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[40839]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:05:18.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[29577]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T09:05:18.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[29577]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T09:05:18.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:18.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[29577]: from='osd.1 v2:172.21.15.79:6805/4182839712' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:05:18.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-mon[29577]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:05:18.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:18 smithi173 ceph-mon[33421]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T09:05:18.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:18 smithi173 ceph-mon[33421]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T09:05:18.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:18.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:18 smithi173 ceph-mon[33421]: from='osd.1 v2:172.21.15.79:6805/4182839712' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:05:18.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:18 smithi173 ceph-mon[33421]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:05:18.989 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1[51422]: 2024-09-17T09:05:18.753+0000 7f2fce690640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T09:05:18.990 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:18 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1[51422]: 2024-09-17T09:05:18.756+0000 7f2fc5c9a640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:05:18.993 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:05:19.519 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[40839]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:05:19.519 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[40839]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T09:05:19.519 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:19.519 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:19.519 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[40839]: from='osd.1 ' entity='osd.1' 2024-09-17T09:05:19.520 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[40839]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:19.520 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[29577]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:05:19.520 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[29577]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T09:05:19.520 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:19.520 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:19.520 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[29577]: from='osd.1 ' entity='osd.1' 2024-09-17T09:05:19.520 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:19 smithi079 ceph-mon[29577]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:19.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:19 smithi173 ceph-mon[33421]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:05:19.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:19 smithi173 ceph-mon[33421]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T09:05:19.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:19 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:19.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:19 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:19.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:19 smithi173 ceph-mon[33421]: from='osd.1 ' entity='osd.1' 2024-09-17T09:05:19.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:19 smithi173 ceph-mon[33421]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: purged_snaps scrub starts 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: purged_snaps scrub ok 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: osd.1 v2:172.21.15.79:6805/4182839712 boot 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: Detected new or changed devices on smithi079 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi079 to 8032M 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: purged_snaps scrub starts 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: purged_snaps scrub ok 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: osd.1 v2:172.21.15.79:6805/4182839712 boot 2024-09-17T09:05:20.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: Detected new or changed devices on smithi079 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi079 to 8032M 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:20.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:20 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: purged_snaps scrub starts 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: purged_snaps scrub ok 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: osd.1 v2:172.21.15.79:6805/4182839712 boot 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: Detected new or changed devices on smithi079 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:20.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:20.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi079 to 8032M 2024-09-17T09:05:20.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:20.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:20.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:20.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:20 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:21.136 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-17T09:05:21.229 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-17T09:05:21.243 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-09-17T09:05:21.244 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-09-17T09:05:21.244 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125165 s, 838 MB/s 2024-09-17T09:05:21.244 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-09-17T09:05:21.812 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:21 smithi079 ceph-mon[29577]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T09:05:21.812 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:21 smithi079 ceph-mon[29577]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:21.813 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:21 smithi079 ceph-mon[40839]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T09:05:21.813 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:21 smithi079 ceph-mon[40839]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:21.814 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_2 2024-09-17T09:05:22.012 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:05:22.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:21 smithi173 ceph-mon[33421]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T09:05:22.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:21 smithi173 ceph-mon[33421]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:24.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:24 smithi079 ceph-mon[40839]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:24.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:05:24.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:05:24.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:24.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:24 smithi079 ceph-mon[29577]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:24.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:05:24.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:05:24.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:24.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:24 smithi173 ceph-mon[33421]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:24.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:05:24.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:05:24.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:25.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:25 smithi079 ceph-mon[29577]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:05:25.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:25 smithi079 ceph-mon[40839]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:05:25.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:25 smithi173 ceph-mon[33421]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:05:26.316 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[29577]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:26.317 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1950875148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9"}]: dispatch 2024-09-17T09:05:26.317 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9"}]: dispatch 2024-09-17T09:05:26.317 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9"}]': finished 2024-09-17T09:05:26.317 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[29577]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T09:05:26.317 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:26.317 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[40839]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:26.317 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1950875148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9"}]: dispatch 2024-09-17T09:05:26.317 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9"}]: dispatch 2024-09-17T09:05:26.317 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9"}]': finished 2024-09-17T09:05:26.318 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[40839]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T09:05:26.318 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:26.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:26 smithi173 ceph-mon[33421]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:26.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:26 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1950875148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9"}]: dispatch 2024-09-17T09:05:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:26 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9"}]: dispatch 2024-09-17T09:05:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:26 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9"}]': finished 2024-09-17T09:05:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:26 smithi173 ceph-mon[33421]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T09:05:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:27.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:27 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/276531568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:05:27.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:27 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/276531568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:05:27.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:27 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/276531568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:05:28.449 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:28 smithi079 ceph-mon[29577]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:28.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:28 smithi079 ceph-mon[40839]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:28.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:28 smithi173 ceph-mon[33421]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:30.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:30 smithi173 ceph-mon[33421]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:30.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:30 smithi079 ceph-mon[29577]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:30.522 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:30 smithi079 ceph-mon[40839]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:32.186 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:32 smithi079 ceph-mon[29577]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:32.186 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:05:32.187 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:32.187 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:32 smithi079 ceph-mon[40839]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:32.187 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:05:32.187 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:32.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:32 smithi173 ceph-mon[33421]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:32.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:05:32.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:33.355 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:33 smithi079 ceph-mon[29577]: Deploying daemon osd.2 on smithi079 2024-09-17T09:05:33.355 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:33 smithi079 ceph-mon[40839]: Deploying daemon osd.2 on smithi079 2024-09-17T09:05:33.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:33 smithi173 ceph-mon[33421]: Deploying daemon osd.2 on smithi079 2024-09-17T09:05:34.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:34 smithi079 ceph-mon[29577]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:34.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:34 smithi079 ceph-mon[40839]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:34.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:34 smithi173 ceph-mon[33421]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:36.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:36 smithi173 ceph-mon[33421]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:36.516 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:36 smithi079 ceph-mon[29577]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:36.516 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:36 smithi079 ceph-mon[40839]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:37.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:37 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:37.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:37 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:37.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:37 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:37.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:37 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:37.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:37 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:37.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:37 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:37.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:37 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:37.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:37 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:37.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:37 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:38.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:38 smithi079 ceph-mon[29577]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:38.400 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:38 smithi079 ceph-mon[40839]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:38.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:38 smithi173 ceph-mon[33421]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:39.606 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 2 on host 'smithi079' 2024-09-17T09:05:40.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:40.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:40.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:40.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:05:40.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:40.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.197 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:40.197 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:40.197 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:40.197 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.197 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:05:40.197 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:40.197 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.198 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.216 DEBUG:teuthology.orchestra.run.smithi079:osd.2> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.2.service 2024-09-17T09:05:40.218 INFO:tasks.cephadm:Deploying osd.3 on smithi079 with /dev/vg_nvme/lv_1... 2024-09-17T09:05:40.219 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-17T09:05:40.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:05:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:05:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:40.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:41.040 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:05:41.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:41 smithi173 ceph-mon[33421]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T09:05:41.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:41 smithi173 ceph-mon[33421]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T09:05:41.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:41 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:41.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:41 smithi173 ceph-mon[33421]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:05:41.535 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-mon[29577]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T09:05:41.535 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-mon[29577]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T09:05:41.535 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:41.535 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-mon[29577]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:05:41.535 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-mon[40839]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T09:05:41.536 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-mon[40839]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T09:05:41.536 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:41.536 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-mon[40839]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:05:41.791 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2[57821]: 2024-09-17T09:05:41.682+0000 7f8659a5c640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T09:05:41.791 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:41 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2[57821]: 2024-09-17T09:05:41.687+0000 7f865187d640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:05:42.357 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[40839]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:42.357 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[40839]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:05:42.357 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[40839]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T09:05:42.357 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:42.357 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:42.357 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[40839]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' 2024-09-17T09:05:42.357 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[29577]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:42.358 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[29577]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:05:42.358 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[29577]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T09:05:42.358 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:42.358 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:42.358 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:42 smithi079 ceph-mon[29577]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' 2024-09-17T09:05:42.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:42 smithi173 ceph-mon[33421]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:42.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:42 smithi173 ceph-mon[33421]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:05:42.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:42 smithi173 ceph-mon[33421]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T09:05:42.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:42 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:42.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:42 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:42.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:42 smithi173 ceph-mon[33421]: from='osd.2 v2:172.21.15.79:6809/4069279605' entity='osd.2' 2024-09-17T09:05:43.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[29577]: purged_snaps scrub starts 2024-09-17T09:05:43.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[29577]: purged_snaps scrub ok 2024-09-17T09:05:43.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:43.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[29577]: osd.2 v2:172.21.15.79:6809/4069279605 boot 2024-09-17T09:05:43.541 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[29577]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T09:05:43.541 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:43.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[40839]: purged_snaps scrub starts 2024-09-17T09:05:43.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[40839]: purged_snaps scrub ok 2024-09-17T09:05:43.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:43.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[40839]: osd.2 v2:172.21.15.79:6809/4069279605 boot 2024-09-17T09:05:43.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[40839]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T09:05:43.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:43 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:43.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:43 smithi173 ceph-mon[33421]: purged_snaps scrub starts 2024-09-17T09:05:43.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:43 smithi173 ceph-mon[33421]: purged_snaps scrub ok 2024-09-17T09:05:43.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:43 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:43.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:43 smithi173 ceph-mon[33421]: osd.2 v2:172.21.15.79:6809/4069279605 boot 2024-09-17T09:05:43.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:43 smithi173 ceph-mon[33421]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T09:05:43.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:43 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:05:44.114 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-17T09:05:44.198 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-17T09:05:44.212 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-09-17T09:05:44.212 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-09-17T09:05:44.212 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122338 s, 857 MB/s 2024-09-17T09:05:44.212 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: Detected new or changed devices on smithi079 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi079 to 5354M 2024-09-17T09:05:44.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: Detected new or changed devices on smithi079 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.523 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi079 to 5354M 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T09:05:44.524 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:05:44.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:05:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:05:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: Detected new or changed devices on smithi079 2024-09-17T09:05:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:05:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi079 to 5354M 2024-09-17T09:05:44.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:44.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:05:44.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:44.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T09:05:44.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T09:05:44.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:05:44.852 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_1 2024-09-17T09:05:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:05:45.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:45 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T09:05:45.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:45 smithi079 ceph-mon[29577]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T09:05:45.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:45 smithi079 ceph-mon[29577]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:45.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:45 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T09:05:45.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:45 smithi079 ceph-mon[40839]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T09:05:45.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:45 smithi079 ceph-mon[40839]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:46.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:45 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T09:05:46.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:45 smithi173 ceph-mon[33421]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T09:05:46.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:45 smithi173 ceph-mon[33421]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:46.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:05:45 smithi079 sudo[62274]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T09:05:46.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:05:45 smithi079 sudo[62274]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:46.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:05:45 smithi079 sudo[62274]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:46.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:05:45 smithi079 sudo[62274]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:46.409 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62278]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T09:05:46.409 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62278]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:46.409 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62278]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:46.409 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62278]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:46.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62315]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T09:05:46.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62315]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:46.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62315]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:46.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62315]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:46.960 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:46 smithi079 ceph-mon[29577]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T09:05:46.961 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:46 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:46.961 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:46 smithi079 ceph-mon[40839]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T09:05:46.961 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:46 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:46.961 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62351]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T09:05:46.961 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62351]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:46.961 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62351]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:46.961 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62351]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:47.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:46 smithi173 ceph-mon[33421]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T09:05:47.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:46 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:47.232 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62422]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T09:05:47.232 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62422]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:47.232 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62422]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:47.232 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:46 smithi079 sudo[62422]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:47.486 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62479]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T09:05:47.486 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62479]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:47.487 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62479]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:47.487 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62479]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:47.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62520]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T09:05:47.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62520]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:47.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62520]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:47.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62520]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:48.031 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:47 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:05:48.031 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:47 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:05:48.031 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:47 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:48.031 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:47 smithi079 ceph-mon[40839]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:05:48.031 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62523]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T09:05:48.032 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62523]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:48.032 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62523]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:48.032 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 sudo[62523]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:48.032 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:05:48.032 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:05:48.032 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:48.032 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:47 smithi079 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:05:48.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:47 smithi173 sudo[35989]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T09:05:48.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:47 smithi173 sudo[35989]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:48.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:47 smithi173 sudo[35989]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:48.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 sudo[35989]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:48.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 sudo[35992]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T09:05:48.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 sudo[35992]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:48.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 sudo[35992]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:48.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 sudo[35992]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:48.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:05:48.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:05:48.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:48.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:05:48.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 sudo[62582]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T09:05:48.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 sudo[62582]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:48.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 sudo[62582]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:48.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 sudo[62582]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:49.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:05:49.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 sudo[62624]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 sudo[62624]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 sudo[62624]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 sudo[62624]: pam_unix(sudo:session): session closed for user root 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:05:49.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:05:49.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:48 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:05:49.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:05:49.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:05:49.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:05:49.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:48 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:05:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[40839]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:05:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[40839]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:05:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/345714764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b2fe76a-f374-4129-b6a4-225adf1bb221"}]: dispatch 2024-09-17T09:05:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/345714764' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b2fe76a-f374-4129-b6a4-225adf1bb221"}]': finished 2024-09-17T09:05:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[40839]: osdmap e22: 4 total, 3 up, 4 in 2024-09-17T09:05:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:05:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[40839]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1512198140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:05:50.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:05:50.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:05:50.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/345714764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b2fe76a-f374-4129-b6a4-225adf1bb221"}]: dispatch 2024-09-17T09:05:50.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/345714764' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b2fe76a-f374-4129-b6a4-225adf1bb221"}]': finished 2024-09-17T09:05:50.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[29577]: osdmap e22: 4 total, 3 up, 4 in 2024-09-17T09:05:50.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:05:50.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[29577]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:50.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:49 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1512198140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:05:50.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:49 smithi173 ceph-mon[33421]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:05:50.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:49 smithi173 ceph-mon[33421]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:05:50.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:49 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/345714764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b2fe76a-f374-4129-b6a4-225adf1bb221"}]: dispatch 2024-09-17T09:05:50.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:49 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/345714764' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b2fe76a-f374-4129-b6a4-225adf1bb221"}]': finished 2024-09-17T09:05:50.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:49 smithi173 ceph-mon[33421]: osdmap e22: 4 total, 3 up, 4 in 2024-09-17T09:05:50.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:49 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:05:50.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:49 smithi173 ceph-mon[33421]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:50.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:49 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1512198140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:05:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:52 smithi079 ceph-mon[29577]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:52.490 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:52 smithi079 ceph-mon[40839]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:52.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:52 smithi173 ceph-mon[33421]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:54.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:54 smithi079 ceph-mon[29577]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:54.311 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:54 smithi079 ceph-mon[40839]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:54.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:54 smithi173 ceph-mon[33421]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:55.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:05:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:55.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:05:55.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:55.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:05:55.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:05:56.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:56 smithi079 ceph-mon[29577]: Deploying daemon osd.3 on smithi079 2024-09-17T09:05:56.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:56 smithi079 ceph-mon[29577]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:56.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:56 smithi079 ceph-mon[40839]: Deploying daemon osd.3 on smithi079 2024-09-17T09:05:56.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:56 smithi079 ceph-mon[40839]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:56.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:56 smithi173 ceph-mon[33421]: Deploying daemon osd.3 on smithi079 2024-09-17T09:05:56.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:56 smithi173 ceph-mon[33421]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:58.177 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:05:58 smithi079 ceph-mon[29577]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:58.177 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:05:58 smithi079 ceph-mon[40839]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:05:58.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:05:58 smithi173 ceph-mon[33421]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:00.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:00 smithi079 ceph-mon[29577]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:00.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:00.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:00.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:00 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:00.439 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:00 smithi079 ceph-mon[40839]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:00.439 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:00 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:00.439 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:00 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:00.440 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:00 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:00.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:00 smithi173 ceph-mon[33421]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:00.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:00 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:00.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:00 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:00.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:00 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:02.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:02 smithi173 ceph-mon[33421]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:02.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:02 smithi079 ceph-mon[29577]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:02.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:02 smithi079 ceph-mon[40839]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:03.272 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 3 on host 'smithi079' 2024-09-17T09:06:03.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='osd.3 v2:172.21.15.79:6813/3572077598' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='osd.3 v2:172.21.15.79:6813/3572077598' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:03 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='osd.3 v2:172.21.15.79:6813/3572077598' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:03 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:03.906 DEBUG:teuthology.orchestra.run.smithi079:osd.3> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.3.service 2024-09-17T09:06:03.908 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/vg_nvme/lv_4... 2024-09-17T09:06:03.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-17T09:06:04.113 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:06:04.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:04 smithi173 ceph-mon[33421]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T09:06:04.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:04 smithi173 ceph-mon[33421]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T09:06:04.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:04 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:04.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:04 smithi173 ceph-mon[33421]: from='osd.3 v2:172.21.15.79:6813/3572077598' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:06:04.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:04 smithi173 ceph-mon[33421]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:06:04.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[29577]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T09:06:04.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[29577]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T09:06:04.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:04.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[29577]: from='osd.3 v2:172.21.15.79:6813/3572077598' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:06:04.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[29577]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:06:04.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[40839]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T09:06:04.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[40839]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T09:06:04.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:04.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[40839]: from='osd.3 v2:172.21.15.79:6813/3572077598' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:06:04.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-mon[40839]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-17T09:06:05.252 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3[64427]: 2024-09-17T09:06:04.970+0000 7fa9e4538640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T09:06:05.252 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:06:04 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3[64427]: 2024-09-17T09:06:04.976+0000 7fa9dc359640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:06:05.733 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:05 smithi173 ceph-mon[33421]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:06:05.733 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:05 smithi173 ceph-mon[33421]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T09:06:05.733 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:05 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:05.733 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:05 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:05.733 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:05 smithi173 ceph-mon[33421]: from='osd.3 ' entity='osd.3' 2024-09-17T09:06:05.733 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:05 smithi173 ceph-mon[33421]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:05.733 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:05 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:05.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[29577]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:06:05.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[29577]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T09:06:05.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:05.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:05.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[29577]: from='osd.3 ' entity='osd.3' 2024-09-17T09:06:05.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[29577]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:05.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:05.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[40839]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-17T09:06:05.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[40839]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T09:06:05.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:05.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:05.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[40839]: from='osd.3 ' entity='osd.3' 2024-09-17T09:06:05.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[40839]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:06:05.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:05 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:05.889 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-17T09:06:05.974 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-17T09:06:05.990 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-09-17T09:06:05.990 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-09-17T09:06:05.991 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135379 s, 775 MB/s 2024-09-17T09:06:05.991 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-09-17T09:06:06.540 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:06 smithi173 ceph-mon[33421]: purged_snaps scrub starts 2024-09-17T09:06:06.540 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:06 smithi173 ceph-mon[33421]: purged_snaps scrub ok 2024-09-17T09:06:06.540 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:06 smithi173 ceph-mon[33421]: osd.3 v2:172.21.15.79:6813/3572077598 boot 2024-09-17T09:06:06.540 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:06 smithi173 ceph-mon[33421]: osdmap e25: 4 total, 4 up, 4 in 2024-09-17T09:06:06.540 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:06 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:06.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-09-17T09:06:06.739 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:06:06.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[29577]: purged_snaps scrub starts 2024-09-17T09:06:06.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[29577]: purged_snaps scrub ok 2024-09-17T09:06:06.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[29577]: osd.3 v2:172.21.15.79:6813/3572077598 boot 2024-09-17T09:06:06.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[29577]: osdmap e25: 4 total, 4 up, 4 in 2024-09-17T09:06:06.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:06.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[40839]: purged_snaps scrub starts 2024-09-17T09:06:06.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[40839]: purged_snaps scrub ok 2024-09-17T09:06:06.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[40839]: osd.3 v2:172.21.15.79:6813/3572077598 boot 2024-09-17T09:06:06.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[40839]: osdmap e25: 4 total, 4 up, 4 in 2024-09-17T09:06:06.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:06 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:06:07.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: Detected new or changed devices on smithi079 2024-09-17T09:06:07.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi079 to 4016M 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[29577]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: Detected new or changed devices on smithi079 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi079 to 4016M 2024-09-17T09:06:07.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:07.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:07.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:07 smithi079 ceph-mon[40839]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:07.918 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: Detected new or changed devices on smithi079 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi079 to 4016M 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:07.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:07.920 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:07 smithi173 ceph-mon[33421]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:08.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:08 smithi079 ceph-mon[29577]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T09:06:08.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:08 smithi079 ceph-mon[40839]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T09:06:08.948 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:08 smithi173 ceph-mon[33421]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T09:06:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[29577]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:06:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:06:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:06:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:09.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[29577]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[40839]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:06:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:06:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:06:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:09 smithi079 ceph-mon[40839]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:09.949 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:09 smithi173 ceph-mon[33421]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:06:09.949 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:09 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:06:09.950 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:09 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:06:09.950 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:09 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:09.950 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:09 smithi173 ceph-mon[33421]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:11.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:10 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/3044803537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40024528-daa8-4ba6-98b2-cf5bb6c4e0f2"}]: dispatch 2024-09-17T09:06:11.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:10 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40024528-daa8-4ba6-98b2-cf5bb6c4e0f2"}]: dispatch 2024-09-17T09:06:11.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:10 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40024528-daa8-4ba6-98b2-cf5bb6c4e0f2"}]': finished 2024-09-17T09:06:11.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:10 smithi173 ceph-mon[33421]: osdmap e27: 5 total, 4 up, 5 in 2024-09-17T09:06:11.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:10 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:11.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/3044803537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40024528-daa8-4ba6-98b2-cf5bb6c4e0f2"}]: dispatch 2024-09-17T09:06:11.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40024528-daa8-4ba6-98b2-cf5bb6c4e0f2"}]: dispatch 2024-09-17T09:06:11.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40024528-daa8-4ba6-98b2-cf5bb6c4e0f2"}]': finished 2024-09-17T09:06:11.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[40839]: osdmap e27: 5 total, 4 up, 5 in 2024-09-17T09:06:11.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:11.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/3044803537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40024528-daa8-4ba6-98b2-cf5bb6c4e0f2"}]: dispatch 2024-09-17T09:06:11.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40024528-daa8-4ba6-98b2-cf5bb6c4e0f2"}]: dispatch 2024-09-17T09:06:11.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40024528-daa8-4ba6-98b2-cf5bb6c4e0f2"}]': finished 2024-09-17T09:06:11.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[29577]: osdmap e27: 5 total, 4 up, 5 in 2024-09-17T09:06:11.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:10 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:12.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:11 smithi173 ceph-mon[33421]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:12.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:11 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/3831672228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:06:12.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:11 smithi079 ceph-mon[29577]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:12.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:11 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/3831672228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:06:12.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:11 smithi079 ceph-mon[40839]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:12.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:11 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/3831672228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:06:14.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:14 smithi173 ceph-mon[33421]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:14.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:14 smithi079 ceph-mon[29577]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:14.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:14 smithi079 ceph-mon[40839]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:16.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:16 smithi079 ceph-mon[29577]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:16.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:16 smithi079 ceph-mon[40839]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:16.441 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:16 smithi173 ceph-mon[33421]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:17.422 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:06:17.422 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:17.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:06:17.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:17.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:06:17.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:18.468 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:18 smithi173 ceph-mon[33421]: Deploying daemon osd.4 on smithi173 2024-09-17T09:06:18.468 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:18 smithi173 ceph-mon[33421]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:18.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:18 smithi079 ceph-mon[29577]: Deploying daemon osd.4 on smithi173 2024-09-17T09:06:18.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:18 smithi079 ceph-mon[29577]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:18.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:18 smithi079 ceph-mon[40839]: Deploying daemon osd.4 on smithi173 2024-09-17T09:06:18.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:18 smithi079 ceph-mon[40839]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:20.335 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:20 smithi173 ceph-mon[33421]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:20.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:20 smithi079 ceph-mon[29577]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:20 smithi079 ceph-mon[40839]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:22.399 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:22 smithi173 ceph-mon[33421]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:22.400 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:22 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:22.400 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:22 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:22.400 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:22 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:22.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:22 smithi079 ceph-mon[29577]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:22.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:22 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:22.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:22 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:22.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:22 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:22.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:22 smithi079 ceph-mon[40839]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:22.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:22 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:22.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:22 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:22.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:22 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:24.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:24 smithi173 ceph-mon[33421]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:24.558 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-09-17T09:06:24.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:24 smithi079 ceph-mon[29577]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:24.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:24 smithi079 ceph-mon[40839]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:25.499 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.4.service 2024-09-17T09:06:25.502 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_3... 2024-09-17T09:06:25.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-17T09:06:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:06:25.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:25 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:25.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:25 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:25.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:25 smithi173 ceph-mon[33421]: from='osd.4 v2:172.21.15.173:6800/316529671' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:06:25.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:25 smithi173 ceph-mon[33421]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:06:25.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:25 smithi173 ceph-mon[33421]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:25.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:25.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:25.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[40839]: from='osd.4 v2:172.21.15.173:6800/316529671' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:06:25.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[40839]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:06:25.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[40839]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:25.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:25.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:25.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[29577]: from='osd.4 v2:172.21.15.173:6800/316529671' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:06:25.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[29577]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:06:25.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:25 smithi079 ceph-mon[29577]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:26.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[29577]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T09:06:26.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[29577]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[29577]: from='osd.4 v2:172.21.15.173:6800/316529671' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[29577]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[40839]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[40839]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[40839]: from='osd.4 v2:172.21.15.173:6800/316529671' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:26.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[40839]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:26.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:26.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:26.904 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:26 smithi173 ceph-mon[33421]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T09:06:26.904 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:26 smithi173 ceph-mon[33421]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T09:06:26.904 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:26.904 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:26 smithi173 ceph-mon[33421]: from='osd.4 v2:172.21.15.173:6800/316529671' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:26.904 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:26 smithi173 ceph-mon[33421]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:26.904 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:26.904 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:26.904 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:06:26 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:06:26.574+0000 7f73b4c77640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T09:06:26.905 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:06:26 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:06:26.577+0000 7f73b0aa0640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:06:27.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:27 smithi173 ceph-mon[33421]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:06:27.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:27 smithi173 ceph-mon[33421]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T09:06:27.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:27.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:27.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:27 smithi173 ceph-mon[33421]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:27.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[29577]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:06:27.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[29577]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T09:06:27.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:27.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:27.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[29577]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:27.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[40839]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:06:27.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[40839]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T09:06:27.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:27.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:27.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:27 smithi079 ceph-mon[40839]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:06:28.092 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-17T09:06:28.184 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-17T09:06:28.199 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-09-17T09:06:28.199 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-09-17T09:06:28.199 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125539 s, 835 MB/s 2024-09-17T09:06:28.199 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-09-17T09:06:28.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:28 smithi173 ceph-mon[33421]: purged_snaps scrub starts 2024-09-17T09:06:28.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:28 smithi173 ceph-mon[33421]: purged_snaps scrub ok 2024-09-17T09:06:28.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:28 smithi173 ceph-mon[33421]: osd.4 v2:172.21.15.173:6800/316529671 boot 2024-09-17T09:06:28.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:28 smithi173 ceph-mon[33421]: osdmap e30: 5 total, 5 up, 5 in 2024-09-17T09:06:28.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:28 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:28.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:28 smithi173 ceph-mon[33421]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T09:06:28.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[40839]: purged_snaps scrub starts 2024-09-17T09:06:28.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[40839]: purged_snaps scrub ok 2024-09-17T09:06:28.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[40839]: osd.4 v2:172.21.15.173:6800/316529671 boot 2024-09-17T09:06:28.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[40839]: osdmap e30: 5 total, 5 up, 5 in 2024-09-17T09:06:28.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:28.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[40839]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T09:06:28.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[29577]: purged_snaps scrub starts 2024-09-17T09:06:28.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[29577]: purged_snaps scrub ok 2024-09-17T09:06:28.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[29577]: osd.4 v2:172.21.15.173:6800/316529671 boot 2024-09-17T09:06:28.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[29577]: osdmap e30: 5 total, 5 up, 5 in 2024-09-17T09:06:28.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:06:28.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:28 smithi079 ceph-mon[29577]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T09:06:29.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-09-17T09:06:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:06:29.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:29 smithi079 ceph-mon[40839]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:29.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:29 smithi079 ceph-mon[40839]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T09:06:29.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:29 smithi079 ceph-mon[29577]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:29.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:29 smithi079 ceph-mon[29577]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T09:06:29.901 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:29 smithi173 ceph-mon[33421]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:29.901 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:29 smithi173 ceph-mon[33421]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T09:06:31.687 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:31 smithi173 ceph-mon[33421]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T09:06:31.687 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:31 smithi173 ceph-mon[33421]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-17T09:06:31.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:31 smithi079 ceph-mon[29577]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T09:06:31.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:31 smithi079 ceph-mon[29577]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-17T09:06:31.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:31 smithi079 ceph-mon[40839]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T09:06:31.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:31 smithi079 ceph-mon[40839]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-17T09:06:32.716 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T09:06:32.716 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: Detected new or changed devices on smithi173 2024-09-17T09:06:32.716 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.716 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.716 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi173 to 17088M 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:06:32.717 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:32 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: Detected new or changed devices on smithi173 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi173 to 17088M 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:32.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: Detected new or changed devices on smithi173 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi173 to 17088M 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:06:32.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:32 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:33.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:33 smithi173 ceph-mon[33421]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-09-17T09:06:33.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:33 smithi079 ceph-mon[29577]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-09-17T09:06:33.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:33 smithi079 ceph-mon[40839]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-09-17T09:06:34.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/305428117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bc50575-e098-48bb-b2b8-ceab32659051"}]: dispatch 2024-09-17T09:06:34.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bc50575-e098-48bb-b2b8-ceab32659051"}]: dispatch 2024-09-17T09:06:34.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bc50575-e098-48bb-b2b8-ceab32659051"}]': finished 2024-09-17T09:06:34.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[29577]: osdmap e35: 6 total, 5 up, 6 in 2024-09-17T09:06:34.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:34.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/305428117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bc50575-e098-48bb-b2b8-ceab32659051"}]: dispatch 2024-09-17T09:06:34.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bc50575-e098-48bb-b2b8-ceab32659051"}]: dispatch 2024-09-17T09:06:34.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bc50575-e098-48bb-b2b8-ceab32659051"}]': finished 2024-09-17T09:06:34.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[40839]: osdmap e35: 6 total, 5 up, 6 in 2024-09-17T09:06:34.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:34 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:35.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:34 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/305428117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bc50575-e098-48bb-b2b8-ceab32659051"}]: dispatch 2024-09-17T09:06:35.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:34 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bc50575-e098-48bb-b2b8-ceab32659051"}]: dispatch 2024-09-17T09:06:35.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:34 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bc50575-e098-48bb-b2b8-ceab32659051"}]': finished 2024-09-17T09:06:35.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:34 smithi173 ceph-mon[33421]: osdmap e35: 6 total, 5 up, 6 in 2024-09-17T09:06:35.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:34 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:35.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:06:35 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:06:35.061+0000 7f6be577d640 -1 calc_pg_upmaps abort due to max <= 0 2024-09-17T09:06:35.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:35 smithi079 ceph-mon[29577]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-17T09:06:35.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:35 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/280968702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:06:35.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:35 smithi079 ceph-mon[40839]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-17T09:06:35.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:35 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/280968702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:06:36.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:35 smithi173 ceph-mon[33421]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-17T09:06:36.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:35 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/280968702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:06:38.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:38 smithi173 ceph-mon[33421]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:38.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:38 smithi079 ceph-mon[29577]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:38.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:38 smithi079 ceph-mon[40839]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:40.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:40 smithi173 ceph-mon[33421]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:40.567 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:40 smithi079 ceph-mon[29577]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:40.567 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:40 smithi079 ceph-mon[40839]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:41.433 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:41 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:06:41.434 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:41 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:41.573 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:41 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:06:41.573 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:41 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:41.573 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:41 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:06:41.573 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:41 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:42.429 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:42 smithi173 ceph-mon[33421]: Deploying daemon osd.5 on smithi173 2024-09-17T09:06:42.429 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:42 smithi173 ceph-mon[33421]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:42.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:42 smithi079 ceph-mon[29577]: Deploying daemon osd.5 on smithi173 2024-09-17T09:06:42.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:42 smithi079 ceph-mon[29577]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:42.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:42 smithi079 ceph-mon[40839]: Deploying daemon osd.5 on smithi173 2024-09-17T09:06:42.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:42 smithi079 ceph-mon[40839]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:44.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:44 smithi173 ceph-mon[33421]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:44.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:44 smithi079 ceph-mon[29577]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:44.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:44 smithi079 ceph-mon[40839]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:46.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:46 smithi173 ceph-mon[33421]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:46.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:46 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:46.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:46 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:46.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:46 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:46.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:46 smithi079 ceph-mon[40839]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:46.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:46 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:46.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:46 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:46.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:46 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:46.482 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:46 smithi079 ceph-mon[29577]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:46.482 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:46 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:46.482 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:46 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:46.482 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:46 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:48.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:48 smithi173 ceph-mon[33421]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:48.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:48 smithi079 ceph-mon[29577]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:48.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:48 smithi079 ceph-mon[40839]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:49.306 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-09-17T09:06:49.616 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.616 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.616 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:49.616 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:49.616 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.616 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='osd.5 v2:172.21.15.173:6804/2138335494' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:06:49.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:06:49.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:49.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:49.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:49 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='osd.5 v2:172.21.15.173:6804/2138335494' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:49.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:49.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='osd.5 v2:172.21.15.173:6804/2138335494' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:06:49.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:06:49.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:49.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:06:49.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:49 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:49.941 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.5.service 2024-09-17T09:06:49.943 INFO:tasks.cephadm:Deploying osd.6 on smithi173 with /dev/vg_nvme/lv_2... 2024-09-17T09:06:49.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-17T09:06:50.686 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:06:50.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:50 smithi173 ceph-mon[33421]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T09:06:50.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:50 smithi173 ceph-mon[33421]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T09:06:50.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:50 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:50.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:50 smithi173 ceph-mon[33421]: from='osd.5 v2:172.21.15.173:6804/2138335494' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:50.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:50 smithi173 ceph-mon[33421]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:50.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[29577]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T09:06:50.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[29577]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T09:06:50.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:50.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[29577]: from='osd.5 v2:172.21.15.173:6804/2138335494' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:50.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[29577]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:50.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[40839]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T09:06:50.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[40839]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T09:06:50.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:50.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[40839]: from='osd.5 v2:172.21.15.173:6804/2138335494' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:50.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:50 smithi079 ceph-mon[40839]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:06:51.257 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:06:50 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:06:50.922+0000 7f4637350640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T09:06:51.257 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:06:50 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:06:50.926+0000 7f462f171640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:06:51.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:51 smithi173 ceph-mon[33421]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:06:51.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:51 smithi173 ceph-mon[33421]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T09:06:51.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:51 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:51.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:51 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:51.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:51 smithi173 ceph-mon[33421]: from='osd.5 ' entity='osd.5' 2024-09-17T09:06:51.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:51 smithi173 ceph-mon[33421]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:51.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:51 smithi173 ceph-mon[33421]: osd.5 v2:172.21.15.173:6804/2138335494 boot 2024-09-17T09:06:51.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[29577]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:06:51.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[29577]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T09:06:51.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:51.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:51.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[29577]: from='osd.5 ' entity='osd.5' 2024-09-17T09:06:51.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[29577]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:51.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[29577]: osd.5 v2:172.21.15.173:6804/2138335494 boot 2024-09-17T09:06:51.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[40839]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:06:51.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[40839]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T09:06:51.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:51.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:51.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[40839]: from='osd.5 ' entity='osd.5' 2024-09-17T09:06:51.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[40839]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:06:51.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:51 smithi079 ceph-mon[40839]: osd.5 v2:172.21.15.173:6804/2138335494 boot 2024-09-17T09:06:52.689 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:52 smithi173 ceph-mon[33421]: purged_snaps scrub starts 2024-09-17T09:06:52.689 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:52 smithi173 ceph-mon[33421]: purged_snaps scrub ok 2024-09-17T09:06:52.689 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:52 smithi173 ceph-mon[33421]: osdmap e38: 6 total, 6 up, 6 in 2024-09-17T09:06:52.689 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:52 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:52.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:52 smithi079 ceph-mon[40839]: purged_snaps scrub starts 2024-09-17T09:06:52.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:52 smithi079 ceph-mon[40839]: purged_snaps scrub ok 2024-09-17T09:06:52.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:52 smithi079 ceph-mon[40839]: osdmap e38: 6 total, 6 up, 6 in 2024-09-17T09:06:52.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:52 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:52.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:52 smithi079 ceph-mon[29577]: purged_snaps scrub starts 2024-09-17T09:06:52.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:52 smithi079 ceph-mon[29577]: purged_snaps scrub ok 2024-09-17T09:06:52.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:52 smithi079 ceph-mon[29577]: osdmap e38: 6 total, 6 up, 6 in 2024-09-17T09:06:52.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:52 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:06:53.388 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-17T09:06:53.476 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-17T09:06:53.490 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-09-17T09:06:53.491 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-09-17T09:06:53.491 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012846 s, 816 MB/s 2024-09-17T09:06:53.491 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-09-17T09:06:53.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:53 smithi173 ceph-mon[33421]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T09:06:53.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:53 smithi173 ceph-mon[33421]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:06:53.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:53 smithi173 ceph-mon[33421]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T09:06:53.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:53 smithi079 ceph-mon[40839]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T09:06:53.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:53 smithi079 ceph-mon[40839]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:06:53.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:53 smithi079 ceph-mon[40839]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T09:06:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:53 smithi079 ceph-mon[29577]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T09:06:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:53 smithi079 ceph-mon[29577]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:06:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:53 smithi079 ceph-mon[29577]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T09:06:54.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-09-17T09:06:54.895 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:06:55.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: Detected new or changed devices on smithi173 2024-09-17T09:06:55.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:55.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:55.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi173 to 8544M 2024-09-17T09:06:55.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:55.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:55.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:54 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: Detected new or changed devices on smithi173 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi173 to 8544M 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: Detected new or changed devices on smithi173 2024-09-17T09:06:55.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:55.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:06:55.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi173 to 8544M 2024-09-17T09:06:55.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:55.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:55.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:06:55.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:54 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:06:56.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:55 smithi173 ceph-mon[33421]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T09:06:56.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:55 smithi173 ceph-mon[33421]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:06:56.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:55 smithi079 ceph-mon[29577]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T09:06:56.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:55 smithi079 ceph-mon[29577]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:06:56.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:55 smithi079 ceph-mon[40839]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T09:06:56.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:55 smithi079 ceph-mon[40839]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:06:57.242 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:56 smithi173 ceph-mon[33421]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T09:06:57.242 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:56 smithi173 ceph-mon[33421]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T09:06:57.242 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:56 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:06:57.242 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:56 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:06:57.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:56 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:57.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[29577]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T09:06:57.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[29577]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T09:06:57.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:06:57.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:06:57.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:57.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[40839]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T09:06:57.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[40839]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T09:06:57.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:06:57.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:06:57.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:56 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:06:58.241 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:57 smithi173 ceph-mon[33421]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:06:58.241 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:57 smithi173 ceph-mon[33421]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:06:58.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:57 smithi079 ceph-mon[40839]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:06:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:57 smithi079 ceph-mon[40839]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:06:58.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:57 smithi079 ceph-mon[29577]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:06:58.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:57 smithi079 ceph-mon[29577]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:06:59.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:58 smithi173 ceph-mon[33421]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T09:06:59.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:58 smithi173 ceph-mon[33421]: Cluster is now healthy 2024-09-17T09:06:59.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:58 smithi079 ceph-mon[29577]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T09:06:59.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:58 smithi079 ceph-mon[29577]: Cluster is now healthy 2024-09-17T09:06:59.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:58 smithi079 ceph-mon[40839]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T09:06:59.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:58 smithi079 ceph-mon[40839]: Cluster is now healthy 2024-09-17T09:07:00.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:59 smithi173 ceph-mon[33421]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:00.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:59 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/150879437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e4eab1-7edb-4ffb-8573-6adedb7842cb"}]: dispatch 2024-09-17T09:07:00.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:59 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e4eab1-7edb-4ffb-8573-6adedb7842cb"}]: dispatch 2024-09-17T09:07:00.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:59 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65e4eab1-7edb-4ffb-8573-6adedb7842cb"}]': finished 2024-09-17T09:07:00.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:59 smithi173 ceph-mon[33421]: osdmap e43: 7 total, 6 up, 7 in 2024-09-17T09:07:00.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:59 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:00.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:06:59 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/203651693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:07:00.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[40839]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/150879437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e4eab1-7edb-4ffb-8573-6adedb7842cb"}]: dispatch 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e4eab1-7edb-4ffb-8573-6adedb7842cb"}]: dispatch 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65e4eab1-7edb-4ffb-8573-6adedb7842cb"}]': finished 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[40839]: osdmap e43: 7 total, 6 up, 7 in 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/203651693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[29577]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/150879437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e4eab1-7edb-4ffb-8573-6adedb7842cb"}]: dispatch 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e4eab1-7edb-4ffb-8573-6adedb7842cb"}]: dispatch 2024-09-17T09:07:00.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65e4eab1-7edb-4ffb-8573-6adedb7842cb"}]': finished 2024-09-17T09:07:00.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[29577]: osdmap e43: 7 total, 6 up, 7 in 2024-09-17T09:07:00.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:00.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:06:59 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/203651693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:07:02.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:02 smithi173 ceph-mon[33421]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-17T09:07:02.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:02 smithi079 ceph-mon[29577]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-17T09:07:02.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:02 smithi079 ceph-mon[40839]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-17T09:07:04.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:04 smithi173 ceph-mon[33421]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-17T09:07:04.414 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:04 smithi079 ceph-mon[29577]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-17T09:07:04.414 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:04 smithi079 ceph-mon[40839]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-17T09:07:06.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:06 smithi173 ceph-mon[33421]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-09-17T09:07:06.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:06 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:07:06.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:06 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:06.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:06 smithi173 ceph-mon[33421]: Deploying daemon osd.6 on smithi173 2024-09-17T09:07:06.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:06 smithi079 ceph-mon[29577]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-09-17T09:07:06.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:06 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:07:06.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:06 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:06.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:06 smithi079 ceph-mon[29577]: Deploying daemon osd.6 on smithi173 2024-09-17T09:07:06.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:06 smithi079 ceph-mon[40839]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-09-17T09:07:06.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:06 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:07:06.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:06 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:06.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:06 smithi079 ceph-mon[40839]: Deploying daemon osd.6 on smithi173 2024-09-17T09:07:08.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:08 smithi173 ceph-mon[33421]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:08.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:08 smithi079 ceph-mon[29577]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:08.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:08 smithi079 ceph-mon[40839]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:10.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:10 smithi173 ceph-mon[33421]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:10.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:10 smithi079 ceph-mon[29577]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:10.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:10 smithi079 ceph-mon[40839]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:11.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:11 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:11.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:11 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:11.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:11 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:11.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:11 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:11.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:11 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:11.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:11 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:11.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:11 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:11.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:11 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:11.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:11 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:12.377 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:12 smithi173 ceph-mon[33421]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:12.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:12 smithi079 ceph-mon[29577]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:12.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:12 smithi079 ceph-mon[40839]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:13.763 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 6 on host 'smithi173' 2024-09-17T09:07:14.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='osd.6 v2:172.21.15.173:6808/3945569006' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:13 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='osd.6 v2:172.21.15.173:6808/3945569006' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='osd.6 v2:172.21.15.173:6808/3945569006' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:13 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:14.390 DEBUG:teuthology.orchestra.run.smithi173:osd.6> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.6.service 2024-09-17T09:07:14.393 INFO:tasks.cephadm:Deploying osd.7 on smithi173 with /dev/vg_nvme/lv_1... 2024-09-17T09:07:14.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-17T09:07:15.197 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:07:15.610 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:15 smithi173 ceph-mon[33421]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T09:07:15.610 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:15 smithi173 ceph-mon[33421]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T09:07:15.610 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:15 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:15.610 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:15 smithi173 ceph-mon[33421]: from='osd.6 v2:172.21.15.173:6808/3945569006' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:15.610 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:15 smithi173 ceph-mon[33421]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:15.610 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:15 smithi173 ceph-mon[33421]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:15.611 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:07:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:07:15.466+0000 7f7af9656640 -1 osd.6 0 waiting for initial osdmap 2024-09-17T09:07:15.611 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:07:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:07:15.472+0000 7f7af547f640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:07:15.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[29577]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[29577]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[29577]: from='osd.6 v2:172.21.15.173:6808/3945569006' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[29577]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[29577]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[40839]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[40839]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[40839]: from='osd.6 v2:172.21.15.173:6808/3945569006' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[40839]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:15.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:15 smithi079 ceph-mon[40839]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:16.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:16 smithi173 ceph-mon[33421]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:07:16.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:16 smithi173 ceph-mon[33421]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T09:07:16.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:16 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:16.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:16 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:16.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:16 smithi079 ceph-mon[29577]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:07:16.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:16 smithi079 ceph-mon[29577]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T09:07:16.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:16 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:16.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:16 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:16.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:16 smithi079 ceph-mon[40839]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:07:16.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:16 smithi079 ceph-mon[40839]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T09:07:16.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:16 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:16.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:16 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:17.596 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:17 smithi173 ceph-mon[33421]: purged_snaps scrub starts 2024-09-17T09:07:17.596 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:17 smithi173 ceph-mon[33421]: purged_snaps scrub ok 2024-09-17T09:07:17.596 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:17.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:17 smithi173 ceph-mon[33421]: osd.6 v2:172.21.15.173:6808/3945569006 boot 2024-09-17T09:07:17.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:17 smithi173 ceph-mon[33421]: osdmap e46: 7 total, 7 up, 7 in 2024-09-17T09:07:17.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:17 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:17.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:17 smithi173 ceph-mon[33421]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:17.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:17 smithi173 ceph-mon[33421]: osdmap e47: 7 total, 7 up, 7 in 2024-09-17T09:07:17.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[29577]: purged_snaps scrub starts 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[29577]: purged_snaps scrub ok 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[29577]: osd.6 v2:172.21.15.173:6808/3945569006 boot 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[29577]: osdmap e46: 7 total, 7 up, 7 in 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[29577]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[29577]: osdmap e47: 7 total, 7 up, 7 in 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[40839]: purged_snaps scrub starts 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[40839]: purged_snaps scrub ok 2024-09-17T09:07:17.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:17.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[40839]: osd.6 v2:172.21.15.173:6808/3945569006 boot 2024-09-17T09:07:17.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[40839]: osdmap e46: 7 total, 7 up, 7 in 2024-09-17T09:07:17.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:07:17.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[40839]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:07:17.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:17 smithi079 ceph-mon[40839]: osdmap e47: 7 total, 7 up, 7 in 2024-09-17T09:07:18.577 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: Detected new or changed devices on smithi173 2024-09-17T09:07:18.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:18.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:18.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:18.581 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.581 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:18.581 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:18.581 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.581 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:18 smithi173 ceph-mon[33421]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T09:07:18.712 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-17T09:07:18.809 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-17T09:07:18.825 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-09-17T09:07:18.825 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-09-17T09:07:18.825 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133218 s, 787 MB/s 2024-09-17T09:07:18.826 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-09-17T09:07:18.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: Detected new or changed devices on smithi173 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[29577]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T09:07:18.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: Detected new or changed devices on smithi173 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:18 smithi079 ceph-mon[40839]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T09:07:19.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-09-17T09:07:19.667 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:07:19.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:19 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi173 to 5696M 2024-09-17T09:07:19.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:19 smithi173 ceph-mon[33421]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:19.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:19 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi173 to 5696M 2024-09-17T09:07:19.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:19 smithi079 ceph-mon[29577]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:19.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:19 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi173 to 5696M 2024-09-17T09:07:19.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:19 smithi079 ceph-mon[40839]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:20.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:20 smithi173 ceph-mon[33421]: osdmap e49: 7 total, 7 up, 7 in 2024-09-17T09:07:20.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:20 smithi079 ceph-mon[29577]: osdmap e49: 7 total, 7 up, 7 in 2024-09-17T09:07:20.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:20 smithi079 ceph-mon[40839]: osdmap e49: 7 total, 7 up, 7 in 2024-09-17T09:07:21.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:21 smithi173 ceph-mon[33421]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T09:07:21.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:21 smithi173 ceph-mon[33421]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:21.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:21 smithi079 ceph-mon[29577]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T09:07:21.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:21 smithi079 ceph-mon[29577]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:21.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:21 smithi079 ceph-mon[40839]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T09:07:21.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:21 smithi079 ceph-mon[40839]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:22.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:22 smithi079 ceph-mon[29577]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:07:22.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:22 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:07:22.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:22 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:07:22.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:22 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:22.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:22 smithi079 ceph-mon[40839]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:07:22.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:22 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:07:22.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:22 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:07:22.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:22 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:22.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:22 smithi173 ceph-mon[33421]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:07:22.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:22 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:07:22.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:22 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:07:22.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:22 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:23.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:23 smithi079 ceph-mon[29577]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-17T09:07:23.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:23 smithi079 ceph-mon[40839]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-17T09:07:24.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:23 smithi173 ceph-mon[33421]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-17T09:07:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/660970410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fff27519-acac-46f9-8655-43a510b21470"}]: dispatch 2024-09-17T09:07:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fff27519-acac-46f9-8655-43a510b21470"}]: dispatch 2024-09-17T09:07:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fff27519-acac-46f9-8655-43a510b21470"}]': finished 2024-09-17T09:07:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[29577]: osdmap e51: 8 total, 7 up, 8 in 2024-09-17T09:07:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/660970410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fff27519-acac-46f9-8655-43a510b21470"}]: dispatch 2024-09-17T09:07:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fff27519-acac-46f9-8655-43a510b21470"}]: dispatch 2024-09-17T09:07:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[40839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fff27519-acac-46f9-8655-43a510b21470"}]': finished 2024-09-17T09:07:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[40839]: osdmap e51: 8 total, 7 up, 8 in 2024-09-17T09:07:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:24 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:25.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:24 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/660970410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fff27519-acac-46f9-8655-43a510b21470"}]: dispatch 2024-09-17T09:07:25.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:24 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fff27519-acac-46f9-8655-43a510b21470"}]: dispatch 2024-09-17T09:07:25.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:24 smithi173 ceph-mon[33421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fff27519-acac-46f9-8655-43a510b21470"}]': finished 2024-09-17T09:07:25.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:24 smithi173 ceph-mon[33421]: osdmap e51: 8 total, 7 up, 8 in 2024-09-17T09:07:25.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:24 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:25.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:25 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/554370335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:07:25.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:25 smithi079 ceph-mon[29577]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T09:07:25.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:25 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/554370335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:07:25.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:25 smithi079 ceph-mon[40839]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T09:07:26.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:25 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/554370335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:07:26.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:25 smithi173 ceph-mon[33421]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T09:07:28.462 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:28 smithi173 ceph-mon[33421]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-17T09:07:28.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:28 smithi079 ceph-mon[29577]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-17T09:07:28.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:28 smithi079 ceph-mon[40839]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-17T09:07:30.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:30 smithi173 ceph-mon[33421]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-17T09:07:30.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:30 smithi079 ceph-mon[29577]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-17T09:07:30.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:30 smithi079 ceph-mon[40839]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-17T09:07:31.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:31 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:07:31.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:31 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:31.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:31 smithi173 ceph-mon[33421]: Deploying daemon osd.7 on smithi173 2024-09-17T09:07:31.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:31 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:07:31.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:31 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:31.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:31 smithi079 ceph-mon[29577]: Deploying daemon osd.7 on smithi173 2024-09-17T09:07:31.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:31 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:07:31.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:31 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:31.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:31 smithi079 ceph-mon[40839]: Deploying daemon osd.7 on smithi173 2024-09-17T09:07:32.302 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:32 smithi173 ceph-mon[33421]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T09:07:32.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:32 smithi079 ceph-mon[29577]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T09:07:32.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:32 smithi079 ceph-mon[40839]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T09:07:34.414 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:34 smithi079 ceph-mon[29577]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:34.414 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:34 smithi079 ceph-mon[40839]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:34.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:34 smithi173 ceph-mon[33421]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:36.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:36 smithi173 ceph-mon[33421]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:36.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:36.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:36.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:36 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:36.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:36 smithi079 ceph-mon[40839]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:36.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:36.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:36.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:36 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:36.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:36 smithi079 ceph-mon[29577]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:36.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:36.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:36.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:36 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:38.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:38 smithi173 ceph-mon[33421]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:38.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:38 smithi079 ceph-mon[29577]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:38.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:38 smithi079 ceph-mon[40839]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:38.922 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 7 on host 'smithi173' 2024-09-17T09:07:39.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:39 smithi173 ceph-mon[33421]: from='osd.7 v2:172.21.15.173:6812/3469976893' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:07:39.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:39 smithi173 ceph-mon[33421]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:07:39.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:39.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:39 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:39.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:39 smithi079 ceph-mon[29577]: from='osd.7 v2:172.21.15.173:6812/3469976893' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:07:39.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:39 smithi079 ceph-mon[29577]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:07:39.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:39.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:39 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:39.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:39 smithi079 ceph-mon[40839]: from='osd.7 v2:172.21.15.173:6812/3469976893' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:07:39.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:39 smithi079 ceph-mon[40839]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:07:39.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:39.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:39 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:39.781 DEBUG:teuthology.orchestra.run.smithi173:osd.7> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.7.service 2024-09-17T09:07:39.783 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-17T09:07:39.784 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T09:07:39.996 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:07:40.353 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[29577]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:40.353 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[29577]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T09:07:40.353 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[29577]: osdmap e52: 8 total, 7 up, 8 in 2024-09-17T09:07:40.354 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:40.354 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[29577]: from='osd.7 v2:172.21.15.173:6812/3469976893' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:40.354 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[29577]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:40.354 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:40.354 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:40.354 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[40839]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:40.354 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[40839]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T09:07:40.354 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[40839]: osdmap e52: 8 total, 7 up, 8 in 2024-09-17T09:07:40.354 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:40.355 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[40839]: from='osd.7 v2:172.21.15.173:6812/3469976893' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:40.355 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[40839]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:40.355 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:40.355 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:40 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:40.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-mon[33421]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:40.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-mon[33421]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T09:07:40.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-mon[33421]: osdmap e52: 8 total, 7 up, 8 in 2024-09-17T09:07:40.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:40.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-mon[33421]: from='osd.7 v2:172.21.15.173:6812/3469976893' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:40.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-mon[33421]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-09-17T09:07:40.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:40.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:41.006 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:07:40.611+0000 7f676830e640 -1 osd.7 0 waiting for initial osdmap 2024-09-17T09:07:41.007 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:07:40 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:07:40.619+0000 7f676012f640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:07:41.317 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:41 smithi173 ceph-mon[33421]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:07:41.317 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:41 smithi173 ceph-mon[33421]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T09:07:41.317 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:41 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:41.317 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:41 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:41.317 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:41 smithi173 ceph-mon[33421]: from='osd.7 ' entity='osd.7' 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[29577]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[29577]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[29577]: from='osd.7 ' entity='osd.7' 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[40839]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[40839]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:41.323 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:41 smithi079 ceph-mon[40839]: from='osd.7 ' entity='osd.7' 2024-09-17T09:07:41.713 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:07:42.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[40839]: purged_snaps scrub starts 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[40839]: purged_snaps scrub ok 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[40839]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[40839]: osd.7 v2:172.21.15.173:6812/3469976893 boot 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[40839]: osdmap e54: 8 total, 8 up, 8 in 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3827200123' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[40839]: osdmap e55: 8 total, 8 up, 8 in 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[29577]: purged_snaps scrub starts 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[29577]: purged_snaps scrub ok 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[29577]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:42.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[29577]: osd.7 v2:172.21.15.173:6812/3469976893 boot 2024-09-17T09:07:42.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[29577]: osdmap e54: 8 total, 8 up, 8 in 2024-09-17T09:07:42.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:42.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3827200123' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T09:07:42.363 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:42 smithi079 ceph-mon[29577]: osdmap e55: 8 total, 8 up, 8 in 2024-09-17T09:07:42.405 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1726564061,"num_in_osds":8,"osd_in_since":1726564043,"num_remapped_pgs":1} 2024-09-17T09:07:42.406 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T09:07:42.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:42 smithi173 ceph-mon[33421]: purged_snaps scrub starts 2024-09-17T09:07:42.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:42 smithi173 ceph-mon[33421]: purged_snaps scrub ok 2024-09-17T09:07:42.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:42 smithi173 ceph-mon[33421]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:07:42.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:42 smithi173 ceph-mon[33421]: osd.7 v2:172.21.15.173:6812/3469976893 boot 2024-09-17T09:07:42.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:42 smithi173 ceph-mon[33421]: osdmap e54: 8 total, 8 up, 8 in 2024-09-17T09:07:42.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:42 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:07:42.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:42 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3827200123' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T09:07:42.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:42 smithi173 ceph-mon[33421]: osdmap e55: 8 total, 8 up, 8 in 2024-09-17T09:07:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:07:44.259 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:07:44.259 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":57,"fsid":"721af936-74d3-11ef-bceb-c7b262605968","created":"2024-09-17T09:02:43.785707+0000","modified":"2024-09-17T09:07:43.417350+0000","last_up_change":"2024-09-17T09:07:41.182093+0000","last_in_change":"2024-09-17T09:07:23.929239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T09:05:43.352570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f7bcae75-6877-4f44-ac7f-55d7393f6fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":1619782938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1619782938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":1619782938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":1619782938}]},"public_addr":"172.21.15.79:6801/1619782938","cluster_addr":"172.21.15.79:6802/1619782938","heartbeat_back_addr":"172.21.15.79:6804/1619782938","heartbeat_front_addr":"172.21.15.79:6803/1619782938","state":["exists","up"]},{"osd":1,"uuid":"c0cf41b3-7003-426b-af5f-dcf5d7ee48d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":4182839712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4182839712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":4182839712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":4182839712}]},"public_addr":"172.21.15.79:6805/4182839712","cluster_addr":"172.21.15.79:6806/4182839712","heartbeat_back_addr":"172.21.15.79:6808/4182839712","heartbeat_front_addr":"172.21.15.79:6807/4182839712","state":["exists","up"]},{"osd":2,"uuid":"df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":4069279605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":4069279605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":4069279605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":4069279605}]},"public_addr":"172.21.15.79:6809/4069279605","cluster_addr":"172.21.15.79:6810/4069279605","heartbeat_back_addr":"172.21.15.79:6812/4069279605","heartbeat_front_addr":"172.21.15.79:6811/4069279605","state":["exists","up"]},{"osd":3,"uuid":"7b2fe76a-f374-4129-b6a4-225adf1bb221","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6813","nonce":3572077598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3572077598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3572077598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6815","nonce":3572077598}]},"public_addr":"172.21.15.79:6813/3572077598","cluster_addr":"172.21.15.79:6814/3572077598","heartbeat_back_addr":"172.21.15.79:6816/3572077598","heartbeat_front_addr":"172.21.15.79:6815/3572077598","state":["exists","up"]},{"osd":4,"uuid":"40024528-daa8-4ba6-98b2-cf5bb6c4e0f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":316529671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":316529671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":316529671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":316529671}]},"public_addr":"172.21.15.173:6800/316529671","cluster_addr":"172.21.15.173:6801/316529671","heartbeat_back_addr":"172.21.15.173:6803/316529671","heartbeat_front_addr":"172.21.15.173:6802/316529671","state":["exists","up"]},{"osd":5,"uuid":"0bc50575-e098-48bb-b2b8-ceab32659051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2138335494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":2138335494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":2138335494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2138335494}]},"public_addr":"172.21.15.173:6804/2138335494","cluster_addr":"172.21.15.173:6805/2138335494","heartbeat_back_addr":"172.21.15.173:6807/2138335494","heartbeat_front_addr":"172.21.15.173:6806/2138335494","state":["exists","up"]},{"osd":6,"uuid":"65e4eab1-7edb-4ffb-8573-6adedb7842cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3945569006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":3945569006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":3945569006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3945569006}]},"public_addr":"172.21.15.173:6808/3945569006","cluster_addr":"172.21.15.173:6809/3945569006","heartbeat_back_addr":"172.21.15.173:6811/3945569006","heartbeat_front_addr":"172.21.15.173:6810/3945569006","state":["exists","up"]},{"osd":7,"uuid":"fff27519-acac-46f9-8655-43a510b21470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3469976893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3469976893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":3469976893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3469976893}]},"public_addr":"172.21.15.173:6812/3469976893","cluster_addr":"172.21.15.173:6813/3469976893","heartbeat_back_addr":"172.21.15.173:6815/3469976893","heartbeat_front_addr":"172.21.15.173:6814/3469976893","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:04:55.314052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:05:17.919531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:05:40.510188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:06:03.889478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:06:25.925441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:06:49.907149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:07:14.535301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:07:39.788284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1909914308":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/2798908727":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/2547180548":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/1368847569":"2024-09-18T09:03:12.103315+0000","172.21.15.79:6800/1436651616":"2024-09-18T09:03:12.103315+0000","172.21.15.79:0/471197754":"2024-09-18T09:03:12.103315+0000","172.21.15.79:6800/1365476350":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/3758171704":"2024-09-18T09:03:12.103315+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: Detected new or changed devices on smithi173 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi173 to 4272M 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:44.365 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[40839]: osdmap e57: 8 total, 8 up, 8 in 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: Detected new or changed devices on smithi173 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi173 to 4272M 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:44.367 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T09:07:44.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:44 smithi079 ceph-mon[29577]: osdmap e57: 8 total, 8 up, 8 in 2024-09-17T09:07:44.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: Detected new or changed devices on smithi173 2024-09-17T09:07:44.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi173 to 4272M 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:44.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:44.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:44.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:44.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T09:07:44.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:44 smithi173 ceph-mon[33421]: osdmap e57: 8 total, 8 up, 8 in 2024-09-17T09:07:44.881 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T09:05:43.352570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T09:07:44.882 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-17T09:07:45.078 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:07:45.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:45 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/117499462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:07:45.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:45 smithi079 ceph-mon[40839]: osdmap e58: 8 total, 8 up, 8 in 2024-09-17T09:07:45.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:45 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/117499462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:07:45.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:45 smithi079 ceph-mon[29577]: osdmap e58: 8 total, 8 up, 8 in 2024-09-17T09:07:45.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:45 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/117499462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:07:45.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:45 smithi173 ceph-mon[33421]: osdmap e58: 8 total, 8 up, 8 in 2024-09-17T09:07:46.350 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:46 smithi079 ceph-mon[29577]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-09-17T09:07:46.350 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:46 smithi079 ceph-mon[40839]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-09-17T09:07:46.408 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:46 smithi173 ceph-mon[33421]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-09-17T09:07:46.974 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-09-17T09:07:47.264 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:47 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3942186679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T09:07:47.264 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:47 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3942186679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T09:07:47.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:47 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3942186679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T09:07:47.629 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi079 2024-09-17T09:07:47.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi079=foo.a' 2024-09-17T09:07:47.826 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:07:48.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:48 smithi173 ceph-mon[33421]: pgmap v174: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-17T09:07:48.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:48 smithi079 ceph-mon[29577]: pgmap v174: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-17T09:07:48.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:48 smithi079 ceph-mon[40839]: pgmap v174: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-17T09:07:49.907 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled rgw.foo.a update... 2024-09-17T09:07:50.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:50 smithi173 ceph-mon[33421]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T09:07:50.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:50 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:50.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:50 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:50.548 DEBUG:teuthology.orchestra.run.smithi079:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@rgw.foo.a.service 2024-09-17T09:07:50.551 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi173 2024-09-17T09:07:50.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-17T09:07:50.571 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:50 smithi079 ceph-mon[29577]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T09:07:50.571 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:50 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:50.571 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:50 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:50.571 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:50 smithi079 ceph-mon[40839]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T09:07:50.571 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:50 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:50.571 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:50 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:50.745 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:07:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi079=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:07:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-09-17T09:07:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:07:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:07:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:51.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:51 smithi173 ceph-mon[33421]: Deploying daemon rgw.foo.a on smithi079 2024-09-17T09:07:51.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi079=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[29577]: Deploying daemon rgw.foo.a on smithi079 2024-09-17T09:07:51.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi079=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:07:51.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-09-17T09:07:51.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:51.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:51.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:51.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:07:51.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:07:51.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:51.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:51.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:51 smithi079 ceph-mon[40839]: Deploying daemon rgw.foo.a on smithi079 2024-09-17T09:07:52.360 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:52 smithi079 ceph-mon[29577]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-17T09:07:52.360 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:52 smithi079 ceph-mon[40839]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-17T09:07:52.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:52 smithi173 ceph-mon[33421]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-17T09:07:52.611 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:07:52 smithi079 podman[69488]: 2024-09-17 09:07:52.271818874 +0000 UTC m=+0.015848098 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:07:53.112 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:07:52 smithi079 podman[69488]: 2024-09-17 09:07:52.65389497 +0000 UTC m=+0.397924192 container init f48e7cfad8b622a4f6365db4c8dbdf5dd6035163543b6bb2f064684d186552c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-rgw-foo-a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-17T09:07:53.112 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:07:52 smithi079 podman[69488]: 2024-09-17 09:07:52.6589915 +0000 UTC m=+0.403020704 container start f48e7cfad8b622a4f6365db4c8dbdf5dd6035163543b6bb2f064684d186552c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:07:53.113 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:07:52 smithi079 bash[69488]: f48e7cfad8b622a4f6365db4c8dbdf5dd6035163543b6bb2f064684d186552c4 2024-09-17T09:07:53.113 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:07:52 smithi079 systemd[1]: Started Ceph rgw.foo.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:07:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:pool 'datapool' created 2024-09-17T09:07:53.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:53 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/940664013' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T09:07:53.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:53 smithi173 ceph-mon[33421]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T09:07:53.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:53 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:53 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:53 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:53 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:53 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:53 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:53.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/940664013' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T09:07:53.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T09:07:53.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:53.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/940664013' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T09:07:53.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[40839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T09:07:53.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:53.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:53 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:07:53.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- rbd pool init datapool 2024-09-17T09:07:54.113 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:07:54.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:54 smithi173 ceph-mon[33421]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-09-17T09:07:54.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:54 smithi173 ceph-mon[33421]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:07:54.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:54 smithi173 ceph-mon[33421]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-17T09:07:54.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:54 smithi173 ceph-mon[33421]: osdmap e59: 8 total, 8 up, 8 in 2024-09-17T09:07:54.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:54 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/234896557' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T09:07:54.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:54 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T09:07:54.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[40839]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-09-17T09:07:54.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[40839]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:07:54.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[40839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-17T09:07:54.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[40839]: osdmap e59: 8 total, 8 up, 8 in 2024-09-17T09:07:54.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/234896557' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T09:07:54.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T09:07:54.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[29577]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-09-17T09:07:54.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[29577]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:07:54.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-17T09:07:54.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[29577]: osdmap e59: 8 total, 8 up, 8 in 2024-09-17T09:07:54.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/234896557' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T09:07:54.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:54 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T09:07:55.456 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:55 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T09:07:55.456 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:55 smithi173 ceph-mon[33421]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T09:07:55.456 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:55.457 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:55.457 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:55.457 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:55.457 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:55 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:55.457 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:55 smithi173 ceph-mon[33421]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T09:07:55.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T09:07:55.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[29577]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T09:07:55.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:55.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:55.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:55.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:55.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:55.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[29577]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T09:07:55.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T09:07:55.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[40839]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T09:07:55.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:55.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:55.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:07:55.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:07:55.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:07:55.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:55 smithi079 ceph-mon[40839]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T09:07:56.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: Checking dashboard <-> RGW credentials 2024-09-17T09:07:56.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: pgmap v180: 36 pgs: 7 creating+peering, 28 unknown, 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:07:56.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/234896557' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:07:56.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:07:56.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:07:56.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T09:07:56.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/375977850' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T09:07:56.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T09:07:56.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T09:07:56.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T09:07:56.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-17T09:07:56.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:56 smithi173 ceph-mon[33421]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T09:07:56.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: Checking dashboard <-> RGW credentials 2024-09-17T09:07:56.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: pgmap v180: 36 pgs: 7 creating+peering, 28 unknown, 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/234896557' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/375977850' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[29577]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T09:07:56.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: Checking dashboard <-> RGW credentials 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: pgmap v180: 36 pgs: 7 creating+peering, 28 unknown, 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/234896557' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/375977850' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-17T09:07:56.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:56 smithi079 ceph-mon[40839]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T09:07:57.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:57 smithi173 ceph-mon[33421]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T09:07:57.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:57 smithi173 ceph-mon[33421]: Cluster is now healthy 2024-09-17T09:07:57.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:57 smithi173 ceph-mon[33421]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T09:07:57.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:57 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:07:57.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:57 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/234896557' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:07:57.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:57 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:07:57.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[29577]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T09:07:57.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[29577]: Cluster is now healthy 2024-09-17T09:07:57.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[29577]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T09:07:57.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:07:57.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/234896557' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:07:57.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:07:57.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[40839]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T09:07:57.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[40839]: Cluster is now healthy 2024-09-17T09:07:57.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[40839]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T09:07:57.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:07:57.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/234896557' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:07:57.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:57 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:07:58.606 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:58 smithi173 ceph-mon[33421]: pgmap v183: 68 pgs: 12 creating+peering, 31 unknown, 25 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s 2024-09-17T09:07:58.606 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:58 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T09:07:58.606 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:58 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T09:07:58.606 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:07:58 smithi173 ceph-mon[33421]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T09:07:58.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:58 smithi079 ceph-mon[29577]: pgmap v183: 68 pgs: 12 creating+peering, 31 unknown, 25 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s 2024-09-17T09:07:58.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:58 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T09:07:58.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:58 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T09:07:58.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:07:58 smithi079 ceph-mon[29577]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T09:07:58.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:58 smithi079 ceph-mon[40839]: pgmap v183: 68 pgs: 12 creating+peering, 31 unknown, 25 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s 2024-09-17T09:07:58.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:58 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1258501811' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T09:07:58.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:58 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T09:07:58.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:07:58 smithi079 ceph-mon[40839]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T09:07:58.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.173 --placement '1;smithi173=iscsi.a' 2024-09-17T09:07:59.130 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:08:00.454 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:00 smithi173 ceph-mon[33421]: pgmap v186: 100 pgs: 17 creating+peering, 17 unknown, 66 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-17T09:08:00.455 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:00 smithi173 ceph-mon[33421]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T09:08:00.455 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:00 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3222733679' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.455 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:00 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.455 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:00 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2672758906' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.455 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:00 smithi173 ceph-mon[33421]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[29577]: pgmap v186: 100 pgs: 17 creating+peering, 17 unknown, 66 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-17T09:08:00.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[29577]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T09:08:00.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3222733679' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2672758906' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[29577]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[40839]: pgmap v186: 100 pgs: 17 creating+peering, 17 unknown, 66 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-17T09:08:00.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[40839]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T09:08:00.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3222733679' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2672758906' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:00.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:00 smithi079 ceph-mon[40839]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:08:01.143 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled iscsi.datapool update... 2024-09-17T09:08:01.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T09:08:01.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[29577]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T09:08:01.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[29577]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T09:08:01.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3222733679' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2672758906' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[29577]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:01.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T09:08:01.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[40839]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T09:08:01.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[40839]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T09:08:01.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3222733679' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2672758906' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[40839]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:01 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:01.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:01 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T09:08:01.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:01 smithi173 ceph-mon[33421]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T09:08:01.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:01 smithi173 ceph-mon[33421]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T09:08:01.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:01 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3222733679' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:01 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:01 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2672758906' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:01 smithi173 ceph-mon[33421]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:08:01.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:01 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:01.833 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-17T09:08:01.833 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:08:01.834 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-17T09:08:01.862 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:08:01.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-17T09:08:01.890 DEBUG:teuthology.orchestra.run.smithi173:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@iscsi.iscsi.a.service 2024-09-17T09:08:01.933 INFO:tasks.cephadm:Adding prometheus.a on smithi173 2024-09-17T09:08:01.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi173=a' 2024-09-17T09:08:02.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: pgmap v189: 132 pgs: 1 creating+activating, 12 creating+peering, 42 unknown, 77 active+clean; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-17T09:08:02.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:02.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-09-17T09:08:02.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T09:08:02.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T09:08:02.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T09:08:02.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:02.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:02.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:02.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:08:02.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T09:08:02.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:02 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: pgmap v189: 132 pgs: 1 creating+activating, 12 creating+peering, 42 unknown, 77 active+clean; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:02.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: pgmap v189: 132 pgs: 1 creating+activating, 12 creating+peering, 42 unknown, 77 active+clean; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:02.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:02.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:08:02.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T09:08:02.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:02 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:08:02.708 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:08:03.437 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:03 smithi173 ceph-mon[33421]: Deploying daemon iscsi.iscsi.a on smithi173 2024-09-17T09:08:03.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:03 smithi079 ceph-mon[29577]: Deploying daemon iscsi.iscsi.a on smithi173 2024-09-17T09:08:03.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:03 smithi079 ceph-mon[40839]: Deploying daemon iscsi.iscsi.a on smithi173 2024-09-17T09:08:04.586 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:04 smithi173 ceph-mon[33421]: pgmap v191: 132 pgs: 1 creating+activating, 5 creating+peering, 7 unknown, 119 active+clean; 580 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 5.2 KiB/s wr, 126 op/s 2024-09-17T09:08:04.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:04 smithi079 ceph-mon[29577]: pgmap v191: 132 pgs: 1 creating+activating, 5 creating+peering, 7 unknown, 119 active+clean; 580 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 5.2 KiB/s wr, 126 op/s 2024-09-17T09:08:04.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:04 smithi079 ceph-mon[40839]: pgmap v191: 132 pgs: 1 creating+activating, 5 creating+peering, 7 unknown, 119 active+clean; 580 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 5.2 KiB/s wr, 126 op/s 2024-09-17T09:08:05.100 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled prometheus update... 2024-09-17T09:08:06.170 DEBUG:teuthology.orchestra.run.smithi173:prometheus.a> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-17T09:08:06.173 INFO:tasks.cephadm:Adding node-exporter.a on smithi079 2024-09-17T09:08:06.173 INFO:tasks.cephadm:Adding node-exporter.b on smithi173 2024-09-17T09:08:06.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi079=a;smithi173=b' 2024-09-17T09:08:06.360 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:06 smithi173 ceph-mon[33421]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:06.360 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:06 smithi173 ceph-mon[33421]: Saving service prometheus spec with placement smithi173=a;count:1 2024-09-17T09:08:06.360 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:06 smithi173 ceph-mon[33421]: pgmap v192: 132 pgs: 1 creating+activating, 131 active+clean; 581 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 5.0 KiB/s wr, 137 op/s 2024-09-17T09:08:06.360 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:06 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:06.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:06 smithi079 ceph-mon[40839]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:06.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:06 smithi079 ceph-mon[40839]: Saving service prometheus spec with placement smithi173=a;count:1 2024-09-17T09:08:06.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:06 smithi079 ceph-mon[40839]: pgmap v192: 132 pgs: 1 creating+activating, 131 active+clean; 581 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 5.0 KiB/s wr, 137 op/s 2024-09-17T09:08:06.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:06 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:06.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:06 smithi079 ceph-mon[29577]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:06.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:06 smithi079 ceph-mon[29577]: Saving service prometheus spec with placement smithi173=a;count:1 2024-09-17T09:08:06.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:06 smithi079 ceph-mon[29577]: pgmap v192: 132 pgs: 1 creating+activating, 131 active+clean; 581 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 5.0 KiB/s wr, 137 op/s 2024-09-17T09:08:06.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:06 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:06.619 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:06 smithi173 podman[62625]: 2024-09-17 09:08:06.391524808 +0000 UTC m=+0.145257577 container create 14190907a849c76640fbea0a70a299c57c25cb49cfcf61d627fb454b259cf4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T09:08:06.619 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:06 smithi173 podman[62625]: 2024-09-17 09:08:06.619030063 +0000 UTC m=+0.372762831 container init 14190907a849c76640fbea0a70a299c57c25cb49cfcf61d627fb454b259cf4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T09:08:06.796 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:08:06.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:06 smithi173 podman[62625]: 2024-09-17 09:08:06.623843216 +0000 UTC m=+0.377575986 container start 14190907a849c76640fbea0a70a299c57c25cb49cfcf61d627fb454b259cf4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:08:06.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:06 smithi173 bash[62625]: 14190907a849c76640fbea0a70a299c57c25cb49cfcf61d627fb454b259cf4de 2024-09-17T09:08:06.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:06 smithi173 systemd[1]: Started Ceph iscsi.iscsi.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:08:07.222 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:06 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug Started the configuration object watcher 2024-09-17T09:08:07.222 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:06 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug Checking for config object changes every 1s 2024-09-17T09:08:07.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:06 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug Processing osd blocklist entries for this node 2024-09-17T09:08:07.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug Reading the configuration object to update local LIO configuration 2024-09-17T09:08:07.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug Configuration does not have an entry for this host(smithi173.front.sepia.ceph.com) - nothing to define to LIO 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: * Environment: production 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: Use a production WSGI server instead. 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: * Debug mode: off 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug * Running on all addresses. 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: * Running on all addresses. 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug * Running on http://[fe80::ec4:7aff:fe88:7259]:5000/ (Press CTRL+C to quit) 2024-09-17T09:08:07.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: * Running on http://[fe80::ec4:7aff:fe88:7259]:5000/ (Press CTRL+C to quit) 2024-09-17T09:08:07.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:07.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:07.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:07.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-mon[33421]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-17T09:08:07.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:07.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:08:07.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-mon[33421]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.9 KiB/s wr, 143 op/s 2024-09-17T09:08:07.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:07 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/3404046996' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T09:08:08.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[40839]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[40839]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.9 KiB/s wr, 143 op/s 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/3404046996' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[29577]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-17T09:08:08.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:08.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:08:08.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[29577]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.9 KiB/s wr, 143 op/s 2024-09-17T09:08:08.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:07 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/3404046996' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T09:08:08.683 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled node-exporter update... 2024-09-17T09:08:09.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:08 smithi079 ceph-mon[29577]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T09:08:09.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:08 smithi079 ceph-mon[29577]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T09:08:09.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:08 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:09.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:08 smithi079 ceph-mon[40839]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T09:08:09.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:08 smithi079 ceph-mon[40839]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T09:08:09.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:08 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:09.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:08 smithi173 ceph-mon[33421]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T09:08:09.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:08 smithi173 ceph-mon[33421]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T09:08:09.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:08 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:09.730 DEBUG:teuthology.orchestra.run.smithi079:node-exporter.a> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-17T09:08:09.732 DEBUG:teuthology.orchestra.run.smithi173:node-exporter.b> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-17T09:08:09.735 INFO:tasks.cephadm:Adding alertmanager.a on smithi079 2024-09-17T09:08:09.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi079=a' 2024-09-17T09:08:10.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:09 smithi173 ceph-mon[33421]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi079=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:10.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:09 smithi173 ceph-mon[33421]: Saving service node-exporter spec with placement smithi079=a;smithi173=b;count:2 2024-09-17T09:08:10.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:09 smithi173 ceph-mon[33421]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 4.7 KiB/s wr, 141 op/s 2024-09-17T09:08:10.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:09 smithi079 ceph-mon[29577]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi079=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:10.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:09 smithi079 ceph-mon[29577]: Saving service node-exporter spec with placement smithi079=a;smithi173=b;count:2 2024-09-17T09:08:10.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:09 smithi079 ceph-mon[29577]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 4.7 KiB/s wr, 141 op/s 2024-09-17T09:08:10.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:09 smithi079 ceph-mon[40839]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi079=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:10.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:09 smithi079 ceph-mon[40839]: Saving service node-exporter spec with placement smithi079=a;smithi173=b;count:2 2024-09-17T09:08:10.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:09 smithi079 ceph-mon[40839]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 4.7 KiB/s wr, 141 op/s 2024-09-17T09:08:10.614 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:08:11.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:11 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:11 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:11 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:11 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:08:11.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:11 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:08:11.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:11 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:08:11.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:08:11.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:11 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:12.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:12 smithi173 ceph-mon[33421]: Deploying daemon prometheus.a on smithi173 2024-09-17T09:08:12.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:12 smithi173 ceph-mon[33421]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.9 KiB/s wr, 114 op/s 2024-09-17T09:08:12.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:12 smithi079 ceph-mon[29577]: Deploying daemon prometheus.a on smithi173 2024-09-17T09:08:12.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:12 smithi079 ceph-mon[29577]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.9 KiB/s wr, 114 op/s 2024-09-17T09:08:12.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:12 smithi079 ceph-mon[40839]: Deploying daemon prometheus.a on smithi173 2024-09-17T09:08:12.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:12 smithi079 ceph-mon[40839]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.9 KiB/s wr, 114 op/s 2024-09-17T09:08:12.632 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled alertmanager update... 2024-09-17T09:08:13.674 DEBUG:teuthology.orchestra.run.smithi079:alertmanager.a> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-17T09:08:13.677 INFO:tasks.cephadm:Adding grafana.a on smithi173 2024-09-17T09:08:13.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi173=a' 2024-09-17T09:08:14.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:13 smithi173 ceph-mon[33421]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi079=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:14.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:13 smithi173 ceph-mon[33421]: Saving service alertmanager spec with placement smithi079=a;count:1 2024-09-17T09:08:14.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:13 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:14.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:13 smithi173 ceph-mon[33421]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1.5 KiB/s wr, 52 op/s 2024-09-17T09:08:14.086 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:08:14.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:13 smithi079 ceph-mon[29577]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi079=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:14.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:13 smithi079 ceph-mon[29577]: Saving service alertmanager spec with placement smithi079=a;count:1 2024-09-17T09:08:14.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:13 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:14.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:13 smithi079 ceph-mon[29577]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1.5 KiB/s wr, 52 op/s 2024-09-17T09:08:14.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:13 smithi079 ceph-mon[40839]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi079=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:14.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:13 smithi079 ceph-mon[40839]: Saving service alertmanager spec with placement smithi079=a;count:1 2024-09-17T09:08:14.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:13 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:14.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:13 smithi079 ceph-mon[40839]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1.5 KiB/s wr, 52 op/s 2024-09-17T09:08:16.302 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:16 smithi173 ceph-mon[33421]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 921 B/s wr, 31 op/s 2024-09-17T09:08:16.302 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:16 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:16.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:16 smithi079 ceph-mon[29577]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 921 B/s wr, 31 op/s 2024-09-17T09:08:16.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:16 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:16.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:16 smithi079 ceph-mon[40839]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 921 B/s wr, 31 op/s 2024-09-17T09:08:16.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:16 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:17.021 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled grafana update... 2024-09-17T09:08:17.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:16 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:08:18.261 DEBUG:teuthology.orchestra.run.smithi173:grafana.a> sudo journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@grafana.a.service 2024-09-17T09:08:18.264 INFO:tasks.cephadm:Setting up client nodes... 2024-09-17T09:08:18.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T09:08:18.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:18 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:18.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:18 smithi173 ceph-mon[33421]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:18.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:18 smithi173 ceph-mon[33421]: Saving service grafana spec with placement smithi173=a;count:1 2024-09-17T09:08:18.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:18 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:18.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:18 smithi173 ceph-mon[33421]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-09-17T09:08:18.289 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:18.289 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[29577]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:18.289 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[29577]: Saving service grafana spec with placement smithi173=a;count:1 2024-09-17T09:08:18.289 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:18.289 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[29577]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-09-17T09:08:18.290 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:18.290 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[40839]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:08:18.290 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[40839]: Saving service grafana spec with placement smithi173=a;count:1 2024-09-17T09:08:18.290 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:18.290 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:18 smithi079 ceph-mon[40839]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-09-17T09:08:18.468 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:20.233 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-09-17T09:08:20.233 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAER+lmMUmcDRAAw29Ryxh5zXvuQrHlY0DOGA== 2024-09-17T09:08:20.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:20 smithi173 ceph-mon[33421]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-09-17T09:08:20.538 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:20 smithi079 ceph-mon[29577]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-09-17T09:08:20.538 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:20 smithi079 ceph-mon[40839]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-09-17T09:08:20.777 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-17T09:08:20.777 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-17T09:08:20.777 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T09:08:20.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T09:08:21.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:21 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/4280678451' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:08:21.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:21 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/4280678451' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:08:21.531 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T09:08:21.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:21 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4280678451' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:08:21.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:21 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4280678451' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:08:21.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:21 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/4280678451' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:08:21.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:21 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/4280678451' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:08:22.417 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:22 smithi173 ceph-mon[33421]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:22.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:22 smithi079 ceph-mon[29577]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:22.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:22 smithi079 ceph-mon[40839]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:24.169 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-09-17T09:08:24.169 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAIR+lm4MDBCRAAokHOVwRcC32umDYrtV5kFg== 2024-09-17T09:08:24.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:24 smithi173 ceph-mon[33421]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:24.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:24 smithi173 ceph-mon[33421]: from='client.? 172.21.15.173:0/3925818156' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:08:24.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:24 smithi173 ceph-mon[33421]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:08:24.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:24 smithi173 ceph-mon[33421]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:08:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:24 smithi079 ceph-mon[29577]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:24 smithi079 ceph-mon[29577]: from='client.? 172.21.15.173:0/3925818156' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:08:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:24 smithi079 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:08:24.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:24 smithi079 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:08:24.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:24 smithi079 ceph-mon[40839]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:24.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:24 smithi079 ceph-mon[40839]: from='client.? 172.21.15.173:0/3925818156' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:08:24.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:24 smithi079 ceph-mon[40839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:08:24.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:24 smithi079 ceph-mon[40839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:08:25.116 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T09:08:25.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-17T09:08:25.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-17T09:08:25.158 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T09:08:25.158 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-17T09:08:25.159 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-17T09:08:25.368 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:25.506 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:08:25 smithi173 systemd[1]: Started Ceph prometheus.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:08:26.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[29577]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:26.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:26.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:26.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:26.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T09:08:26.437 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[40839]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:26.437 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:26.437 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:26.438 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:26.438 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T09:08:26.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-mon[33421]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:26.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' 2024-09-17T09:08:26.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T09:08:26.507 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: ignoring --setuser ceph since I am not root 2024-09-17T09:08:26.507 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: ignoring --setgroup ceph since I am not root 2024-09-17T09:08:26.507 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-mgr[35337]: -- 172.21.15.173:0/338912644 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a816a7ad00 con 0x55a816a4f400 2024-09-17T09:08:26.700 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ignoring --setuser ceph since I am not root 2024-09-17T09:08:26.700 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ignoring --setgroup ceph since I am not root 2024-09-17T09:08:26.700 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:26.529+0000 7f953ced6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:08:26.700 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:26.589+0000 7f953ced6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:08:26.828 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:26.532+0000 7f891bbd2040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:08:26.828 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:26.616+0000 7f891bbd2040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:08:26.828 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:26.763+0000 7f891bbd2040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:08:26.828 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:26.827+0000 7f891bbd2040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:08:26.961 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:26.741+0000 7f953ced6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:08:26.961 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:26.806+0000 7f953ced6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:08:26.961 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:26 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:26.960+0000 7f953ced6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:08:27.110 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:26 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:26.971+0000 7f891bbd2040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:08:27.249 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:08:27.250 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:27.107+0000 7f953ced6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:08:27.387 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:27 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:27.110+0000 7f891bbd2040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:08:27.572 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-mon[29577]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T09:08:27.572 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-mon[29577]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T09:08:27.572 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3640797648' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T09:08:27.572 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-mon[40839]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T09:08:27.572 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-mon[40839]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T09:08:27.572 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3640797648' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T09:08:27.573 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:27.428+0000 7f953ced6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:08:27.573 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:27.491+0000 7f953ced6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:08:27.659 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:27 smithi173 ceph-mon[33421]: from='mgr.14150 172.21.15.79:0/1666796222' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T09:08:27.659 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:27 smithi173 ceph-mon[33421]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T09:08:27.659 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:27 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3640797648' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T09:08:27.659 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:27 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:27.415+0000 7f891bbd2040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:08:27.659 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:27 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:27.476+0000 7f891bbd2040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:08:27.660 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:27 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:27.656+0000 7f891bbd2040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:08:27.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:27.675+0000 7f953ced6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:08:27.986 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":93937120}]},"active_addr":"172.21.15.79:6800/93937120","active_change":"2024-09-17T09:03:35.047091+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.79:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3330254457}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3941619260}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":635173210}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":2171668314}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":2901375531}]}]} 2024-09-17T09:08:27.991 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-17T09:08:27.991 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-17T09:08:27.991 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T09:08:28.199 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:28.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:28 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:28.597+0000 7f953ced6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:08:28.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:28 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:28.669+0000 7f953ced6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:08:29.006 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:28 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:28.521+0000 7f891bbd2040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:08:29.006 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:28 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:28.587+0000 7f891bbd2040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:08:29.545 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:29 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:29.287+0000 7f891bbd2040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:08:29.545 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:29 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:29.402+0000 7f891bbd2040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:08:29.611 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:29 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:29.403+0000 7f953ced6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:08:29.611 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:29 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:29.520+0000 7f953ced6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:08:29.820 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:29 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:29.543+0000 7f891bbd2040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:08:29.820 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:29 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:29.604+0000 7f891bbd2040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:08:29.820 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:29 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:29.751+0000 7f891bbd2040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:08:29.954 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:29 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:29.665+0000 7f953ced6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:08:29.954 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:29 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:29.732+0000 7f953ced6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:08:29.954 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:29 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:29.883+0000 7f953ced6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:08:29.954 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:29 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:29.953+0000 7f953ced6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:08:30.055 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:08:30.055 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":68,"fsid":"721af936-74d3-11ef-bceb-c7b262605968","created":"2024-09-17T09:02:43.785707+0000","modified":"2024-09-17T09:08:08.576465+0000","last_up_change":"2024-09-17T09:07:41.182093+0000","last_in_change":"2024-09-17T09:07:23.929239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T09:05:43.352570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-17T09:07:52.764439+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-17T09:07:52.998207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-17T09:07:54.512581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-17T09:07:56.498964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-17T09:07:58.574565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f7bcae75-6877-4f44-ac7f-55d7393f6fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":1619782938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1619782938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":1619782938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":1619782938}]},"public_addr":"172.21.15.79:6801/1619782938","cluster_addr":"172.21.15.79:6802/1619782938","heartbeat_back_addr":"172.21.15.79:6804/1619782938","heartbeat_front_addr":"172.21.15.79:6803/1619782938","state":["exists","up"]},{"osd":1,"uuid":"c0cf41b3-7003-426b-af5f-dcf5d7ee48d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":4182839712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4182839712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":4182839712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":4182839712}]},"public_addr":"172.21.15.79:6805/4182839712","cluster_addr":"172.21.15.79:6806/4182839712","heartbeat_back_addr":"172.21.15.79:6808/4182839712","heartbeat_front_addr":"172.21.15.79:6807/4182839712","state":["exists","up"]},{"osd":2,"uuid":"df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":4069279605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":4069279605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":4069279605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":4069279605}]},"public_addr":"172.21.15.79:6809/4069279605","cluster_addr":"172.21.15.79:6810/4069279605","heartbeat_back_addr":"172.21.15.79:6812/4069279605","heartbeat_front_addr":"172.21.15.79:6811/4069279605","state":["exists","up"]},{"osd":3,"uuid":"7b2fe76a-f374-4129-b6a4-225adf1bb221","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6813","nonce":3572077598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3572077598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3572077598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6815","nonce":3572077598}]},"public_addr":"172.21.15.79:6813/3572077598","cluster_addr":"172.21.15.79:6814/3572077598","heartbeat_back_addr":"172.21.15.79:6816/3572077598","heartbeat_front_addr":"172.21.15.79:6815/3572077598","state":["exists","up"]},{"osd":4,"uuid":"40024528-daa8-4ba6-98b2-cf5bb6c4e0f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":316529671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":316529671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":316529671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":316529671}]},"public_addr":"172.21.15.173:6800/316529671","cluster_addr":"172.21.15.173:6801/316529671","heartbeat_back_addr":"172.21.15.173:6803/316529671","heartbeat_front_addr":"172.21.15.173:6802/316529671","state":["exists","up"]},{"osd":5,"uuid":"0bc50575-e098-48bb-b2b8-ceab32659051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2138335494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":2138335494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":2138335494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2138335494}]},"public_addr":"172.21.15.173:6804/2138335494","cluster_addr":"172.21.15.173:6805/2138335494","heartbeat_back_addr":"172.21.15.173:6807/2138335494","heartbeat_front_addr":"172.21.15.173:6806/2138335494","state":["exists","up"]},{"osd":6,"uuid":"65e4eab1-7edb-4ffb-8573-6adedb7842cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3945569006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":3945569006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":3945569006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3945569006}]},"public_addr":"172.21.15.173:6808/3945569006","cluster_addr":"172.21.15.173:6809/3945569006","heartbeat_back_addr":"172.21.15.173:6811/3945569006","heartbeat_front_addr":"172.21.15.173:6810/3945569006","state":["exists","up"]},{"osd":7,"uuid":"fff27519-acac-46f9-8655-43a510b21470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3469976893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3469976893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":3469976893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3469976893}]},"public_addr":"172.21.15.173:6812/3469976893","cluster_addr":"172.21.15.173:6813/3469976893","heartbeat_back_addr":"172.21.15.173:6815/3469976893","heartbeat_front_addr":"172.21.15.173:6814/3469976893","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:04:55.314052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:05:17.919531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:05:40.510188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:06:03.889478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:06:25.925441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:06:49.907149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:07:14.535301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:07:39.788284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1909914308":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/2798908727":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/2547180548":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/1368847569":"2024-09-18T09:03:12.103315+0000","172.21.15.79:6800/1436651616":"2024-09-18T09:03:12.103315+0000","172.21.15.79:0/471197754":"2024-09-18T09:03:12.103315+0000","172.21.15.79:6800/1365476350":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/3758171704":"2024-09-18T09:03:12.103315+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T09:08:30.096 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:29 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:29.818+0000 7f891bbd2040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:08:30.096 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:29 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:29.877+0000 7f891bbd2040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:08:30.096 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:30.031+0000 7f891bbd2040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:08:30.245 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:30 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1555681598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:08:30.245 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:30 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1555681598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:08:30.246 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:30.014+0000 7f953ced6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:08:30.246 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:30.178+0000 7f953ced6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:08:30.365 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:30 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1555681598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:08:30.366 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:30.094+0000 7f891bbd2040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:08:30.366 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:30.161+0000 7f891bbd2040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:08:30.366 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:30.243+0000 7f891bbd2040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:08:30.366 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:30.305+0000 7f891bbd2040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:08:30.366 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:30.364+0000 7f891bbd2040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:08:30.518 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:30.244+0000 7f953ced6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:08:30.518 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:30.315+0000 7f953ced6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:08:30.519 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:30.399+0000 7f953ced6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:08:30.519 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:30.459+0000 7f953ced6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:08:30.519 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:30.517+0000 7f953ced6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:08:30.763 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-17T09:08:30.763 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T09:08:30.966 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:31.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:31.340+0000 7f953ced6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[29577]: Standby manager daemon x restarted 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[29577]: Standby manager daemon x started 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[40839]: Standby manager daemon x restarted 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[40839]: Standby manager daemon x started 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:08:31.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:08:31.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:08:31.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-mon[33421]: Standby manager daemon x restarted 2024-09-17T09:08:31.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-mon[33421]: Standby manager daemon x started 2024-09-17T09:08:31.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:08:31.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:08:31.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:08:31.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:08:31.507 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:31.141+0000 7f891bbd2040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:08:31.507 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:08:31] ENGINE Bus STARTING 2024-09-17T09:08:31.507 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: CherryPy Checker: 2024-09-17T09:08:31.507 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: The Application mounted at '' has an empty config. 2024-09-17T09:08:31.508 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:08:31.508 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:08:31] ENGINE Serving on http://:::9283 2024-09-17T09:08:31.508 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:08:31 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:08:31] ENGINE Bus STARTED 2024-09-17T09:08:31.783 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:08:31] ENGINE Bus STARTING 2024-09-17T09:08:31.783 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: CherryPy Checker: 2024-09-17T09:08:31.783 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: The Application mounted at '' has an empty config. 2024-09-17T09:08:31.783 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:31.783 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:08:31] ENGINE Serving on http://:::9283 2024-09-17T09:08:31.783 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:08:31] ENGINE Bus STARTED 2024-09-17T09:08:32.446 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-17T09:08:32.446 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: Active manager daemon y restarted 2024-09-17T09:08:32.446 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: Activating manager daemon y 2024-09-17T09:08:32.446 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T09:08:32.446 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: mgrmap e19: y(active, starting, since 0.00926665s), standbys: x 2024-09-17T09:08:32.446 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:08:32.446 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:08:32.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: Manager daemon y is now available 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:08:32.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: Active manager daemon y restarted 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: Activating manager daemon y 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: mgrmap e19: y(active, starting, since 0.00926665s), standbys: x 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:08:32.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: Manager daemon y is now available 2024-09-17T09:08:32.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:08:32.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:32.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:08:32.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:08:32.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:08:32.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:08:32.472 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-17T09:08:32.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: Active manager daemon y restarted 2024-09-17T09:08:32.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: Activating manager daemon y 2024-09-17T09:08:32.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T09:08:32.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: mgrmap e19: y(active, starting, since 0.00926665s), standbys: x 2024-09-17T09:08:32.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:08:32.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:08:32.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:08:32.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:08:32.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:08:32.474 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:08:32.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:08:32.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: Manager daemon y is now available 2024-09-17T09:08:32.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:08:32.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:32.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:08:32.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:08:32.476 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:08:32.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:08:32.945 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":69,"fsid":"721af936-74d3-11ef-bceb-c7b262605968","created":"2024-09-17T09:02:43.785707+0000","modified":"2024-09-17T09:08:31.342191+0000","last_up_change":"2024-09-17T09:07:41.182093+0000","last_in_change":"2024-09-17T09:07:23.929239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T09:05:43.352570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-17T09:07:52.764439+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-17T09:07:52.998207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-17T09:07:54.512581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-17T09:07:56.498964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-17T09:07:58.574565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f7bcae75-6877-4f44-ac7f-55d7393f6fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":1619782938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1619782938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":1619782938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":1619782938}]},"public_addr":"172.21.15.79:6801/1619782938","cluster_addr":"172.21.15.79:6802/1619782938","heartbeat_back_addr":"172.21.15.79:6804/1619782938","heartbeat_front_addr":"172.21.15.79:6803/1619782938","state":["exists","up"]},{"osd":1,"uuid":"c0cf41b3-7003-426b-af5f-dcf5d7ee48d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":4182839712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4182839712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":4182839712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":4182839712}]},"public_addr":"172.21.15.79:6805/4182839712","cluster_addr":"172.21.15.79:6806/4182839712","heartbeat_back_addr":"172.21.15.79:6808/4182839712","heartbeat_front_addr":"172.21.15.79:6807/4182839712","state":["exists","up"]},{"osd":2,"uuid":"df05ffe0-b67d-4b7b-9e5f-cca54ed8fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":4069279605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":4069279605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":4069279605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":4069279605}]},"public_addr":"172.21.15.79:6809/4069279605","cluster_addr":"172.21.15.79:6810/4069279605","heartbeat_back_addr":"172.21.15.79:6812/4069279605","heartbeat_front_addr":"172.21.15.79:6811/4069279605","state":["exists","up"]},{"osd":3,"uuid":"7b2fe76a-f374-4129-b6a4-225adf1bb221","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6813","nonce":3572077598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3572077598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3572077598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6815","nonce":3572077598}]},"public_addr":"172.21.15.79:6813/3572077598","cluster_addr":"172.21.15.79:6814/3572077598","heartbeat_back_addr":"172.21.15.79:6816/3572077598","heartbeat_front_addr":"172.21.15.79:6815/3572077598","state":["exists","up"]},{"osd":4,"uuid":"40024528-daa8-4ba6-98b2-cf5bb6c4e0f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":316529671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":316529671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":316529671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":316529671}]},"public_addr":"172.21.15.173:6800/316529671","cluster_addr":"172.21.15.173:6801/316529671","heartbeat_back_addr":"172.21.15.173:6803/316529671","heartbeat_front_addr":"172.21.15.173:6802/316529671","state":["exists","up"]},{"osd":5,"uuid":"0bc50575-e098-48bb-b2b8-ceab32659051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2138335494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":2138335494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":2138335494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2138335494}]},"public_addr":"172.21.15.173:6804/2138335494","cluster_addr":"172.21.15.173:6805/2138335494","heartbeat_back_addr":"172.21.15.173:6807/2138335494","heartbeat_front_addr":"172.21.15.173:6806/2138335494","state":["exists","up"]},{"osd":6,"uuid":"65e4eab1-7edb-4ffb-8573-6adedb7842cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3945569006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":3945569006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":3945569006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3945569006}]},"public_addr":"172.21.15.173:6808/3945569006","cluster_addr":"172.21.15.173:6809/3945569006","heartbeat_back_addr":"172.21.15.173:6811/3945569006","heartbeat_front_addr":"172.21.15.173:6810/3945569006","state":["exists","up"]},{"osd":7,"uuid":"fff27519-acac-46f9-8655-43a510b21470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3469976893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3469976893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":3469976893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3469976893}]},"public_addr":"172.21.15.173:6812/3469976893","cluster_addr":"172.21.15.173:6813/3469976893","heartbeat_back_addr":"172.21.15.173:6815/3469976893","heartbeat_front_addr":"172.21.15.173:6814/3469976893","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:04:55.314052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:05:17.919531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:05:40.510188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:06:03.889478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:06:25.925441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:06:49.907149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:07:14.535301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:07:39.788284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:6800/93937120":"2024-09-18T09:08:31.342143+0000","172.21.15.79:6800/1365476350":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/2901375531":"2024-09-18T09:08:31.342143+0000","172.21.15.79:6800/1436651616":"2024-09-18T09:03:12.103315+0000","172.21.15.79:0/2171668314":"2024-09-18T09:08:31.342143+0000","172.21.15.79:0/471197754":"2024-09-18T09:03:12.103315+0000","172.21.15.79:0/1368847569":"2024-09-18T09:03:12.103315+0000","172.21.15.79:0/2547180548":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/3330254457":"2024-09-18T09:08:31.342143+0000","172.21.15.79:0/3758171704":"2024-09-18T09:03:12.103315+0000","172.21.15.79:0/2798908727":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/3941619260":"2024-09-18T09:08:31.342143+0000","172.21.15.79:0/1909914308":"2024-09-18T09:03:35.046937+0000","172.21.15.79:0/635173210":"2024-09-18T09:08:31.342143+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T09:08:33.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:32.958+0000 7f949abd0640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:09:08:32] ENGINE Error in HTTPServer.serve 2024-09-17T09:08:33.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: Traceback (most recent call last): 2024-09-17T09:08:33.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:08:33.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self._connections.run(self.expiration_interval) 2024-09-17T09:08:33.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:08:33.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self._run(expiration_interval) 2024-09-17T09:08:33.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:08:33.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:08:33.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:08:33.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:08:33.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:08:33.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: s = self.context.wrap_socket( 2024-09-17T09:08:33.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:08:33.114 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: return self.sslsocket_class._create( 2024-09-17T09:08:33.114 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:08:33.114 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self.do_handshake() 2024-09-17T09:08:33.114 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:08:33.114 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: self._sslobj.do_handshake() 2024-09-17T09:08:33.114 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:08:33.114 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:32 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:08:33.486 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:33 smithi079 ceph-mon[29577]: mgrmap e20: y(active, since 1.01239s), standbys: x 2024-09-17T09:08:33.486 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:33 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:33.486 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:33 smithi079 ceph-mon[29577]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:33.486 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:33 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2432743150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:08:33.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:33 smithi079 ceph-mon[40839]: mgrmap e20: y(active, since 1.01239s), standbys: x 2024-09-17T09:08:33.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:33 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:33.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:33 smithi079 ceph-mon[40839]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:33.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:33 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2432743150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:08:33.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:33 smithi173 ceph-mon[33421]: mgrmap e20: y(active, since 1.01239s), standbys: x 2024-09-17T09:08:33.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:33 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:33.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:33 smithi173 ceph-mon[33421]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:33.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:33 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2432743150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:08:33.753 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:08:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:08:33.776 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-17T09:08:33.776 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-17T09:08:33.777 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-17T09:08:33.777 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-17T09:08:33.778 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-17T09:08:33.778 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-17T09:08:33.778 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-17T09:08:33.779 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-17T09:08:34.323 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:34.356 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[29577]: [17/Sep/2024:09:08:32] ENGINE Bus STARTING 2024-09-17T09:08:34.356 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[29577]: [17/Sep/2024:09:08:32] ENGINE Serving on https://172.21.15.79:7150 2024-09-17T09:08:34.356 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[29577]: [17/Sep/2024:09:08:32] ENGINE Error in HTTPServer.serve 2024-09-17T09:08:34.356 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-17T09:08:34.356 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:08:34.356 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:08:34.356 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:08:34.356 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:08:34.357 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-17T09:08:34.358 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:08:34.358 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[29577]: [17/Sep/2024:09:08:33] ENGINE Serving on http://172.21.15.79:8765 2024-09-17T09:08:34.358 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[29577]: [17/Sep/2024:09:08:33] ENGINE Bus STARTED 2024-09-17T09:08:34.358 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[29577]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:34.358 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[29577]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-17T09:08:34.359 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[40839]: [17/Sep/2024:09:08:32] ENGINE Bus STARTING 2024-09-17T09:08:34.359 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[40839]: [17/Sep/2024:09:08:32] ENGINE Serving on https://172.21.15.79:7150 2024-09-17T09:08:34.359 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[40839]: [17/Sep/2024:09:08:32] ENGINE Error in HTTPServer.serve 2024-09-17T09:08:34.359 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-17T09:08:34.359 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:08:34.359 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:08:34.359 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:08:34.359 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._run(expiration_interval) 2024-09-17T09:08:34.359 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: s = self.context.wrap_socket( 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.sslsocket_class._create( 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.do_handshake() 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._sslobj.do_handshake() 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[40839]: [17/Sep/2024:09:08:33] ENGINE Serving on http://172.21.15.79:8765 2024-09-17T09:08:34.360 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[40839]: [17/Sep/2024:09:08:33] ENGINE Bus STARTED 2024-09-17T09:08:34.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[40839]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:34.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:34 smithi079 ceph-mon[40839]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-17T09:08:34.371 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:34.557 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:34.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:34 smithi173 ceph-mon[33421]: [17/Sep/2024:09:08:32] ENGINE Bus STARTING 2024-09-17T09:08:34.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:34 smithi173 ceph-mon[33421]: [17/Sep/2024:09:08:32] ENGINE Serving on https://172.21.15.79:7150 2024-09-17T09:08:34.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:34 smithi173 ceph-mon[33421]: [17/Sep/2024:09:08:32] ENGINE Error in HTTPServer.serve 2024-09-17T09:08:34.756 INFO:journalctl@ceph.mon.b.smithi173.stdout: Traceback (most recent call last): 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._run(expiration_interval) 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: s = self.context.wrap_socket( 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: return self.sslsocket_class._create( 2024-09-17T09:08:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:08:34.758 INFO:journalctl@ceph.mon.b.smithi173.stdout: self.do_handshake() 2024-09-17T09:08:34.758 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:08:34.758 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._sslobj.do_handshake() 2024-09-17T09:08:34.758 INFO:journalctl@ceph.mon.b.smithi173.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:08:34.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:34 smithi173 ceph-mon[33421]: [17/Sep/2024:09:08:33] ENGINE Serving on http://172.21.15.79:8765 2024-09-17T09:08:34.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:34 smithi173 ceph-mon[33421]: [17/Sep/2024:09:08:33] ENGINE Bus STARTED 2024-09-17T09:08:34.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:34 smithi173 ceph-mon[33421]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:34.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:34 smithi173 ceph-mon[33421]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-17T09:08:35.525 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.525 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi173 to 3760M 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[29577]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.526 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi173 to 3760M 2024-09-17T09:08:35.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.527 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:35 smithi079 ceph-mon[40839]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:35.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi173 to 3760M 2024-09-17T09:08:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:35 smithi173 ceph-mon[33421]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:37.284 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:37.284 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:37.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:37 smithi079 ceph-mon[29577]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-17T09:08:37.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:37 smithi079 ceph-mon[40839]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-17T09:08:37.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:37 smithi173 ceph-mon[33421]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-17T09:08:38.338 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:38 smithi079 ceph-mon[29577]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:38.338 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:38 smithi079 ceph-mon[40839]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:38.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:38 smithi173 ceph-mon[33421]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:08:40.499 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:40 smithi079 ceph-mon[29577]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T09:08:40.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:40 smithi079 ceph-mon[40839]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T09:08:40.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:40 smithi173 ceph-mon[33421]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T09:08:42.276 INFO:teuthology.orchestra.run.smithi079.stdout:128849018908 2024-09-17T09:08:42.277 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-17T09:08:42.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:42 smithi079 ceph-mon[40839]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T09:08:42.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:42 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:42.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:42 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:42.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:42 smithi079 ceph-mon[29577]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T09:08:42.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:42 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:42.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:42 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:42.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:42 smithi173 ceph-mon[33421]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T09:08:42.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:42 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:42.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:42 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:42.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:42 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:08:43.422 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:43.422 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:43.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:43 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:43.760 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:43 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:43.760 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:43 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:43.761 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:43 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:08:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:08:43.781 INFO:teuthology.orchestra.run.smithi079.stdout:163208757272 2024-09-17T09:08:43.782 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-17T09:08:44.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:08:44.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:08:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:44 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:08:44.822 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:08:44.822 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:44.823 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.824 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.824 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.824 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:08:44.824 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:44.824 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:08:44.824 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:44 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:08:44.844 INFO:teuthology.orchestra.run.smithi079.stdout:231928233999 2024-09-17T09:08:44.845 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-17T09:08:45.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi079 to 3760M 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi079 to 3760M 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:45 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi079 to 3760M 2024-09-17T09:08:45.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:08:45.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:08:45.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:08:45.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:08:45.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:08:45.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.760 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.760 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.760 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.760 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:45 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:08:45.809 INFO:teuthology.orchestra.run.smithi079.stdout:55834574890 2024-09-17T09:08:45.810 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-17T09:08:46.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[29577]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:08:46.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[29577]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:08:46.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[29577]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:08:46.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[29577]: Deploying daemon alertmanager.a on smithi079 2024-09-17T09:08:46.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[29577]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:08:46.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[40839]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:08:46.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[40839]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:08:46.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[40839]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:08:46.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[40839]: Deploying daemon alertmanager.a on smithi079 2024-09-17T09:08:46.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:46 smithi079 ceph-mon[40839]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:08:46.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:46 smithi173 ceph-mon[33421]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:08:46.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:46 smithi173 ceph-mon[33421]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:08:46.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:46 smithi173 ceph-mon[33421]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:08:46.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:46 smithi173 ceph-mon[33421]: Deploying daemon alertmanager.a on smithi079 2024-09-17T09:08:46.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:46 smithi173 ceph-mon[33421]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:08:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:47.711 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:47 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:08:47.711 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:47 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:08:47.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:47 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:08:48.547 INFO:teuthology.orchestra.run.smithi079.stdout:34359738416 2024-09-17T09:08:48.548 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-17T09:08:48.714 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:48 smithi079 ceph-mon[29577]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:08:48.714 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:48 smithi079 ceph-mon[40839]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:08:48.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:48 smithi173 ceph-mon[33421]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:08:50.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:50 smithi079 ceph-mon[29577]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:08:50.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:50 smithi079 ceph-mon[40839]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:08:50.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:50 smithi173 ceph-mon[33421]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:08:50.850 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:52.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:52 smithi079 ceph-mon[40839]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:52.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:52 smithi079 ceph-mon[29577]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:52.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:52 smithi173 ceph-mon[33421]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:52.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:08:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:08:53.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:53 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:53.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:53 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:53.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:08:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:08:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:08:53.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:53 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:08:54.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:54 smithi173 ceph-mon[33421]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:54 smithi079 ceph-mon[29577]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:54.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:54 smithi079 ceph-mon[40839]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:55.586 INFO:teuthology.orchestra.run.smithi079.stdout:197568495636 2024-09-17T09:08:55.587 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-17T09:08:55.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:55 smithi173 ceph-mon[33421]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:55.815 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:55 smithi079 ceph-mon[29577]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:55.816 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:55 smithi079 ceph-mon[40839]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:08:55.969 INFO:teuthology.orchestra.run.smithi079.stdout:107374182435 2024-09-17T09:08:55.969 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-17T09:08:56.315 INFO:teuthology.orchestra.run.smithi079.stdout:163208757274 2024-09-17T09:08:56.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2418387202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T09:08:56.816 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2418387202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T09:08:56.851 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2418387202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T09:08:57.307 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:08:57.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:57 smithi173 ceph-mon[33421]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:57.789 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:57 smithi079 ceph-mon[29577]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:57.790 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:57 smithi079 ceph-mon[40839]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:58.044 INFO:teuthology.orchestra.run.smithi079.stdout:231928234002 2024-09-17T09:08:58.789 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:58 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/139687613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T09:08:58.789 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:58 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/139687613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T09:08:58.811 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233999 got 231928234002 for osd.7 2024-09-17T09:08:58.811 DEBUG:teuthology.parallel:result is None 2024-09-17T09:08:58.919 INFO:teuthology.orchestra.run.smithi079.stdout:77309411369 2024-09-17T09:08:58.919 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-17T09:08:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:58 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/139687613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T09:08:59.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:08:59 smithi079 ceph-mon[29577]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:59.704 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:08:59 smithi079 ceph-mon[40839]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:08:59.720 INFO:teuthology.orchestra.run.smithi079.stdout:128849018912 2024-09-17T09:09:00.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:08:59 smithi173 ceph-mon[33421]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:00.530 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757272 got 163208757274 for osd.5 2024-09-17T09:09:00.530 DEBUG:teuthology.parallel:result is None 2024-09-17T09:09:00.751 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:09:00.751 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:09:00.791 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:09:00.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:00 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2381442030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T09:09:00.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:00 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2381442030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T09:09:01.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:00 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2381442030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T09:09:01.488 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:09:01.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:01 smithi079 ceph-mon[29577]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:01.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:01 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:01.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:01 smithi079 ceph-mon[40839]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:01.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:01 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:01.932 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018912 for osd.4 2024-09-17T09:09:01.932 DEBUG:teuthology.parallel:result is None 2024-09-17T09:09:02.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:01 smithi173 ceph-mon[33421]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:02.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:01 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:02.676 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:02 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:02.676 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:02 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:02.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:02 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:02.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:09:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:09:03.786 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:03 smithi079 ceph-mon[29577]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:03.787 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:03 smithi079 ceph-mon[40839]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:03.787 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:09:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:09:04.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:03 smithi173 ceph-mon[33421]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:05.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:05 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4043631257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T09:09:05.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:05 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/4043631257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T09:09:05.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:05 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/4043631257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T09:09:06.459 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:06 smithi079 ceph-mon[29577]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:06.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:06 smithi079 ceph-mon[40839]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:06.497 INFO:teuthology.orchestra.run.smithi079.stdout:197568495639 2024-09-17T09:09:06.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:06 smithi173 ceph-mon[33421]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:06.659 INFO:teuthology.orchestra.run.smithi079.stdout:55834574895 2024-09-17T09:09:06.717 INFO:teuthology.orchestra.run.smithi079.stdout:34359738420 2024-09-17T09:09:07.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:07 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/342959071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T09:09:07.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:07 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3582077102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T09:09:07.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:07 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/342959071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T09:09:07.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:07 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3582077102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T09:09:07.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:07 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/342959071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T09:09:07.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:07 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3582077102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T09:09:07.926 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495639 for osd.6 2024-09-17T09:09:07.926 DEBUG:teuthology.parallel:result is None 2024-09-17T09:09:08.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:08 smithi173 ceph-mon[33421]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:08.533 INFO:teuthology.orchestra.run.smithi079.stdout:107374182439 2024-09-17T09:09:08.533 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:08 smithi079 ceph-mon[29577]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:08.533 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:08 smithi079 ceph-mon[40839]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:09.210 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:09 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1523250465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T09:09:09.210 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:09 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1523250465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T09:09:09.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:09 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1523250465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T09:09:09.803 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738420 for osd.0 2024-09-17T09:09:09.803 DEBUG:teuthology.parallel:result is None 2024-09-17T09:09:10.180 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574890 got 55834574895 for osd.1 2024-09-17T09:09:10.181 DEBUG:teuthology.parallel:result is None 2024-09-17T09:09:10.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:10 smithi173 ceph-mon[33421]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:10.590 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:10 smithi079 ceph-mon[29577]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:10.590 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:10 smithi079 ceph-mon[40839]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:11.524 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:11 smithi079 systemd[1]: Starting Ceph alertmanager.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:09:11.544 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182435 got 107374182439 for osd.3 2024-09-17T09:09:11.544 DEBUG:teuthology.parallel:result is None 2024-09-17T09:09:11.922 INFO:teuthology.orchestra.run.smithi079.stdout:77309411372 2024-09-17T09:09:12.239 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:11 smithi079 podman[76962]: 2024-09-17 09:09:11.956967997 +0000 UTC m=+0.107136415 volume create 869e6ddb23a5adcc0ba379dd47bb998649ad694c2fe4d8fb5cd6ce39244d7910 2024-09-17T09:09:12.239 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:11 smithi079 podman[76962]: 2024-09-17 09:09:11.866522202 +0000 UTC m=+0.016690630 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-17T09:09:12.239 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 podman[76962]: 2024-09-17 09:09:12.041266444 +0000 UTC m=+0.191434856 container create 7d5c3b1d7b0eb38f2ad38c60432596182ad1515e7149dc57540fedb8628257b3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:12.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-mon[29577]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:12.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/239065480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T09:09:12.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-mon[40839]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:12.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/239065480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T09:09:12.613 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 podman[76962]: 2024-09-17 09:09:12.352359046 +0000 UTC m=+0.502527469 container init 7d5c3b1d7b0eb38f2ad38c60432596182ad1515e7149dc57540fedb8628257b3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:12.613 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 podman[76962]: 2024-09-17 09:09:12.357290932 +0000 UTC m=+0.507459344 container start 7d5c3b1d7b0eb38f2ad38c60432596182ad1515e7149dc57540fedb8628257b3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:12.613 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:12.370Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-17T09:09:12.614 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:12.370Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-17T09:09:12.614 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:12.371Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.79 port=9094 2024-09-17T09:09:12.614 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:12.372Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T09:09:12.614 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:12.394Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:09:12.614 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:12.395Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:09:12.615 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:12.397Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-17T09:09:12.615 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:12.397Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-17T09:09:12.615 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 bash[76962]: 7d5c3b1d7b0eb38f2ad38c60432596182ad1515e7149dc57540fedb8628257b3 2024-09-17T09:09:12.615 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:12 smithi079 systemd[1]: Started Ceph alertmanager.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:09:12.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:12 smithi173 ceph-mon[33421]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:12.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:12 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/239065480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T09:09:12.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:09:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:09:13.021 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:12] ENGINE Bus STOPPING 2024-09-17T09:09:13.064 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411372 for osd.2 2024-09-17T09:09:13.064 DEBUG:teuthology.parallel:result is None 2024-09-17T09:09:13.064 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-17T09:09:13.064 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T09:09:13.263 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:09:13.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T09:09:13.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:13] ENGINE Bus STOPPED 2024-09-17T09:09:13.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:13] ENGINE Bus STARTING 2024-09-17T09:09:13.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:13] ENGINE Serving on http://:::9283 2024-09-17T09:09:13.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:13] ENGINE Bus STARTED 2024-09-17T09:09:13.703 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:09:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:09:13.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:13.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:09:13.705 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:13 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:13 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:13.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:13 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:13 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:13 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:13 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:13 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:13 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:13.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:13 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:09:13.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:13 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:14.669 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:14 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:14.372Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000325684s 2024-09-17T09:09:14.670 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:14 smithi079 ceph-mon[29577]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T09:09:14.670 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:14 smithi079 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:09:14.670 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:14 smithi079 ceph-mon[29577]: Deploying daemon grafana.a on smithi173 2024-09-17T09:09:14.670 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:14 smithi079 ceph-mon[29577]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:14.670 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:14 smithi079 ceph-mon[40839]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T09:09:14.671 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:14 smithi079 ceph-mon[40839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:09:14.671 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:14 smithi079 ceph-mon[40839]: Deploying daemon grafana.a on smithi173 2024-09-17T09:09:14.671 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:14 smithi079 ceph-mon[40839]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:14.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:14 smithi173 ceph-mon[33421]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T09:09:14.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:14 smithi173 ceph-mon[33421]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:09:14.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:14 smithi173 ceph-mon[33421]: Deploying daemon grafana.a on smithi173 2024-09-17T09:09:14.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:14 smithi173 ceph-mon[33421]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:14.931 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:09:14.933 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-09-17T09:09:15.581 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-17T09:09:13.366122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":993,"num_read_kb":839,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":597,"ondisk_log_size":597,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":247776,"kb_used_data":7052,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":749713440,"statfs":{"total":767960285184,"available":767706562560,"internally_reserved":0,"allocated":7221248,"data_stored":3886496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":246402617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002473"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583546+0000","last_change":"2024-09-17T09:07:54.251646+0000","last_active":"2024-09-17T09:08:08.583546+0000","last_peered":"2024-09-17T09:08:08.583546+0000","last_clean":"2024-09-17T09:08:08.583546+0000","last_became_active":"2024-09-17T09:07:54.251361+0000","last_became_peered":"2024-09-17T09:07:54.251361+0000","last_unstale":"2024-09-17T09:08:08.583546+0000","last_undegraded":"2024-09-17T09:08:08.583546+0000","last_fullsized":"2024-09-17T09:08:08.583546+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:31.298221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596998+0000","last_change":"2024-09-17T09:07:56.261459+0000","last_active":"2024-09-17T09:08:09.596998+0000","last_peered":"2024-09-17T09:08:09.596998+0000","last_clean":"2024-09-17T09:08:09.596998+0000","last_became_active":"2024-09-17T09:07:56.261158+0000","last_became_peered":"2024-09-17T09:07:56.261158+0000","last_unstale":"2024-09-17T09:08:09.596998+0000","last_undegraded":"2024-09-17T09:08:09.596998+0000","last_fullsized":"2024-09-17T09:08:09.596998+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:22.307427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583644+0000","last_change":"2024-09-17T09:07:58.272427+0000","last_active":"2024-09-17T09:08:08.583644+0000","last_peered":"2024-09-17T09:08:08.583644+0000","last_clean":"2024-09-17T09:08:08.583644+0000","last_became_active":"2024-09-17T09:07:58.272167+0000","last_became_peered":"2024-09-17T09:07:58.272167+0000","last_unstale":"2024-09-17T09:08:08.583644+0000","last_undegraded":"2024-09-17T09:08:08.583644+0000","last_fullsized":"2024-09-17T09:08:08.583644+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:54.033779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597091+0000","last_change":"2024-09-17T09:08:00.278458+0000","last_active":"2024-09-17T09:08:09.597091+0000","last_peered":"2024-09-17T09:08:09.597091+0000","last_clean":"2024-09-17T09:08:09.597091+0000","last_became_active":"2024-09-17T09:08:00.278188+0000","last_became_peered":"2024-09-17T09:08:00.278188+0000","last_unstale":"2024-09-17T09:08:09.597091+0000","last_undegraded":"2024-09-17T09:08:09.597091+0000","last_fullsized":"2024-09-17T09:08:09.597091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:42.543582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597677+0000","last_change":"2024-09-17T09:08:00.413079+0000","last_active":"2024-09-17T09:08:09.597677+0000","last_peered":"2024-09-17T09:08:09.597677+0000","last_clean":"2024-09-17T09:08:09.597677+0000","last_became_active":"2024-09-17T09:08:00.412524+0000","last_became_peered":"2024-09-17T09:08:00.412524+0000","last_unstale":"2024-09-17T09:08:09.597677+0000","last_undegraded":"2024-09-17T09:08:09.597677+0000","last_fullsized":"2024-09-17T09:08:09.597677+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:07.023236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598555+0000","last_change":"2024-09-17T09:07:54.250825+0000","last_active":"2024-09-17T09:08:09.598555+0000","last_peered":"2024-09-17T09:08:09.598555+0000","last_clean":"2024-09-17T09:08:09.598555+0000","last_became_active":"2024-09-17T09:07:54.250607+0000","last_became_peered":"2024-09-17T09:07:54.250607+0000","last_unstale":"2024-09-17T09:08:09.598555+0000","last_undegraded":"2024-09-17T09:08:09.598555+0000","last_fullsized":"2024-09-17T09:08:09.598555+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:42:09.304832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597636+0000","last_change":"2024-09-17T09:07:56.248032+0000","last_active":"2024-09-17T09:08:09.597636+0000","last_peered":"2024-09-17T09:08:09.597636+0000","last_clean":"2024-09-17T09:08:09.597636+0000","last_became_active":"2024-09-17T09:07:56.247566+0000","last_became_peered":"2024-09-17T09:07:56.247566+0000","last_unstale":"2024-09-17T09:08:09.597636+0000","last_undegraded":"2024-09-17T09:08:09.597636+0000","last_fullsized":"2024-09-17T09:08:09.597636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:10:05.758271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596767+0000","last_change":"2024-09-17T09:07:58.269563+0000","last_active":"2024-09-17T09:08:09.596767+0000","last_peered":"2024-09-17T09:08:09.596767+0000","last_clean":"2024-09-17T09:08:09.596767+0000","last_became_active":"2024-09-17T09:07:58.269291+0000","last_became_peered":"2024-09-17T09:07:58.269291+0000","last_unstale":"2024-09-17T09:08:09.596767+0000","last_undegraded":"2024-09-17T09:08:09.596767+0000","last_fullsized":"2024-09-17T09:08:09.596767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:51:14.556764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582720+0000","last_change":"2024-09-17T09:07:54.249208+0000","last_active":"2024-09-17T09:08:08.582720+0000","last_peered":"2024-09-17T09:08:08.582720+0000","last_clean":"2024-09-17T09:08:08.582720+0000","last_became_active":"2024-09-17T09:07:54.248955+0000","last_became_peered":"2024-09-17T09:07:54.248955+0000","last_unstale":"2024-09-17T09:08:08.582720+0000","last_undegraded":"2024-09-17T09:08:08.582720+0000","last_fullsized":"2024-09-17T09:08:08.582720+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:49:57.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597439+0000","last_change":"2024-09-17T09:07:56.252608+0000","last_active":"2024-09-17T09:08:09.597439+0000","last_peered":"2024-09-17T09:08:09.597439+0000","last_clean":"2024-09-17T09:08:09.597439+0000","last_became_active":"2024-09-17T09:07:56.252371+0000","last_became_peered":"2024-09-17T09:07:56.252371+0000","last_unstale":"2024-09-17T09:08:09.597439+0000","last_undegraded":"2024-09-17T09:08:09.597439+0000","last_fullsized":"2024-09-17T09:08:09.597439+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:49.302726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582314+0000","last_change":"2024-09-17T09:07:58.277397+0000","last_active":"2024-09-17T09:08:08.582314+0000","last_peered":"2024-09-17T09:08:08.582314+0000","last_clean":"2024-09-17T09:08:08.582314+0000","last_became_active":"2024-09-17T09:07:58.277138+0000","last_became_peered":"2024-09-17T09:07:58.277138+0000","last_unstale":"2024-09-17T09:08:08.582314+0000","last_undegraded":"2024-09-17T09:08:08.582314+0000","last_fullsized":"2024-09-17T09:08:08.582314+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:40.348729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582553+0000","last_change":"2024-09-17T09:08:00.280989+0000","last_active":"2024-09-17T09:08:08.582553+0000","last_peered":"2024-09-17T09:08:08.582553+0000","last_clean":"2024-09-17T09:08:08.582553+0000","last_became_active":"2024-09-17T09:08:00.280677+0000","last_became_peered":"2024-09-17T09:08:00.280677+0000","last_unstale":"2024-09-17T09:08:08.582553+0000","last_undegraded":"2024-09-17T09:08:08.582553+0000","last_fullsized":"2024-09-17T09:08:08.582553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:09:46.012224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582398+0000","last_change":"2024-09-17T09:07:54.247832+0000","last_active":"2024-09-17T09:08:08.582398+0000","last_peered":"2024-09-17T09:08:08.582398+0000","last_clean":"2024-09-17T09:08:08.582398+0000","last_became_active":"2024-09-17T09:07:54.247560+0000","last_became_peered":"2024-09-17T09:07:54.247560+0000","last_unstale":"2024-09-17T09:08:08.582398+0000","last_undegraded":"2024-09-17T09:08:08.582398+0000","last_fullsized":"2024-09-17T09:08:08.582398+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:17:39.763847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":27,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596936+0000","last_change":"2024-09-17T09:07:56.261559+0000","last_active":"2024-09-17T09:08:09.596936+0000","last_peered":"2024-09-17T09:08:09.596936+0000","last_clean":"2024-09-17T09:08:09.596936+0000","last_became_active":"2024-09-17T09:07:56.261268+0000","last_became_peered":"2024-09-17T09:07:56.261268+0000","last_unstale":"2024-09-17T09:08:09.596936+0000","last_undegraded":"2024-09-17T09:08:09.596936+0000","last_fullsized":"2024-09-17T09:08:09.596936+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:26.542959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596854+0000","last_change":"2024-09-17T09:07:58.280390+0000","last_active":"2024-09-17T09:08:09.596854+0000","last_peered":"2024-09-17T09:08:09.596854+0000","last_clean":"2024-09-17T09:08:09.596854+0000","last_became_active":"2024-09-17T09:07:58.279948+0000","last_became_peered":"2024-09-17T09:07:58.279948+0000","last_unstale":"2024-09-17T09:08:09.596854+0000","last_undegraded":"2024-09-17T09:08:09.596854+0000","last_fullsized":"2024-09-17T09:08:09.596854+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:46.715628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582307+0000","last_change":"2024-09-17T09:08:00.282477+0000","last_active":"2024-09-17T09:08:08.582307+0000","last_peered":"2024-09-17T09:08:08.582307+0000","last_clean":"2024-09-17T09:08:08.582307+0000","last_became_active":"2024-09-17T09:08:00.282170+0000","last_became_peered":"2024-09-17T09:08:00.282170+0000","last_unstale":"2024-09-17T09:08:08.582307+0000","last_undegraded":"2024-09-17T09:08:08.582307+0000","last_fullsized":"2024-09-17T09:08:08.582307+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:57.183905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598016+0000","last_change":"2024-09-17T09:08:00.410688+0000","last_active":"2024-09-17T09:08:09.598016+0000","last_peered":"2024-09-17T09:08:09.598016+0000","last_clean":"2024-09-17T09:08:09.598016+0000","last_became_active":"2024-09-17T09:08:00.410344+0000","last_became_peered":"2024-09-17T09:08:00.410344+0000","last_unstale":"2024-09-17T09:08:09.598016+0000","last_undegraded":"2024-09-17T09:08:09.598016+0000","last_fullsized":"2024-09-17T09:08:09.598016+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:49:48.873644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"60'1","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584690+0000","last_change":"2024-09-17T09:07:54.248415+0000","last_active":"2024-09-17T09:08:08.584690+0000","last_peered":"2024-09-17T09:08:08.584690+0000","last_clean":"2024-09-17T09:08:08.584690+0000","last_became_active":"2024-09-17T09:07:54.248140+0000","last_became_peered":"2024-09-17T09:07:54.248140+0000","last_unstale":"2024-09-17T09:08:08.584690+0000","last_undegraded":"2024-09-17T09:08:08.584690+0000","last_fullsized":"2024-09-17T09:08:08.584690+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:35.469508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597660+0000","last_change":"2024-09-17T09:07:56.257098+0000","last_active":"2024-09-17T09:08:09.597660+0000","last_peered":"2024-09-17T09:08:09.597660+0000","last_clean":"2024-09-17T09:08:09.597660+0000","last_became_active":"2024-09-17T09:07:56.256823+0000","last_became_peered":"2024-09-17T09:07:56.256823+0000","last_unstale":"2024-09-17T09:08:09.597660+0000","last_undegraded":"2024-09-17T09:08:09.597660+0000","last_fullsized":"2024-09-17T09:08:09.597660+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:26:19.900924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583797+0000","last_change":"2024-09-17T09:07:58.278639+0000","last_active":"2024-09-17T09:08:08.583797+0000","last_peered":"2024-09-17T09:08:08.583797+0000","last_clean":"2024-09-17T09:08:08.583797+0000","last_became_active":"2024-09-17T09:07:58.278310+0000","last_became_peered":"2024-09-17T09:07:58.278310+0000","last_unstale":"2024-09-17T09:08:08.583797+0000","last_undegraded":"2024-09-17T09:08:08.583797+0000","last_fullsized":"2024-09-17T09:08:08.583797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:20.244724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598630+0000","last_change":"2024-09-17T09:08:00.411157+0000","last_active":"2024-09-17T09:08:09.598630+0000","last_peered":"2024-09-17T09:08:09.598630+0000","last_clean":"2024-09-17T09:08:09.598630+0000","last_became_active":"2024-09-17T09:08:00.410740+0000","last_became_peered":"2024-09-17T09:08:00.410740+0000","last_unstale":"2024-09-17T09:08:09.598630+0000","last_undegraded":"2024-09-17T09:08:09.598630+0000","last_fullsized":"2024-09-17T09:08:09.598630+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:44:55.092423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597572+0000","last_change":"2024-09-17T09:07:54.239863+0000","last_active":"2024-09-17T09:08:09.597572+0000","last_peered":"2024-09-17T09:08:09.597572+0000","last_clean":"2024-09-17T09:08:09.597572+0000","last_became_active":"2024-09-17T09:07:54.239545+0000","last_became_peered":"2024-09-17T09:07:54.239545+0000","last_unstale":"2024-09-17T09:08:09.597572+0000","last_undegraded":"2024-09-17T09:08:09.597572+0000","last_fullsized":"2024-09-17T09:08:09.597572+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:18:12.769978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":40,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598526+0000","last_change":"2024-09-17T09:07:56.257316+0000","last_active":"2024-09-17T09:08:09.598526+0000","last_peered":"2024-09-17T09:08:09.598526+0000","last_clean":"2024-09-17T09:08:09.598526+0000","last_became_active":"2024-09-17T09:07:56.257088+0000","last_became_peered":"2024-09-17T09:07:56.257088+0000","last_unstale":"2024-09-17T09:08:09.598526+0000","last_undegraded":"2024-09-17T09:08:09.598526+0000","last_fullsized":"2024-09-17T09:08:09.598526+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:37:26.810507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596971+0000","last_change":"2024-09-17T09:07:58.274279+0000","last_active":"2024-09-17T09:08:09.596971+0000","last_peered":"2024-09-17T09:08:09.596971+0000","last_clean":"2024-09-17T09:08:09.596971+0000","last_became_active":"2024-09-17T09:07:58.273485+0000","last_became_peered":"2024-09-17T09:07:58.273485+0000","last_unstale":"2024-09-17T09:08:09.596971+0000","last_undegraded":"2024-09-17T09:08:09.596971+0000","last_fullsized":"2024-09-17T09:08:09.596971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:39.456511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597336+0000","last_change":"2024-09-17T09:08:00.277959+0000","last_active":"2024-09-17T09:08:09.597336+0000","last_peered":"2024-09-17T09:08:09.597336+0000","last_clean":"2024-09-17T09:08:09.597336+0000","last_became_active":"2024-09-17T09:08:00.277650+0000","last_became_peered":"2024-09-17T09:08:00.277650+0000","last_unstale":"2024-09-17T09:08:09.597336+0000","last_undegraded":"2024-09-17T09:08:09.597336+0000","last_fullsized":"2024-09-17T09:08:09.597336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:57.057584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":24,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583808+0000","last_change":"2024-09-17T09:07:54.254871+0000","last_active":"2024-09-17T09:08:08.583808+0000","last_peered":"2024-09-17T09:08:08.583808+0000","last_clean":"2024-09-17T09:08:08.583808+0000","last_became_active":"2024-09-17T09:07:54.254555+0000","last_became_peered":"2024-09-17T09:07:54.254555+0000","last_unstale":"2024-09-17T09:08:08.583808+0000","last_undegraded":"2024-09-17T09:08:08.583808+0000","last_fullsized":"2024-09-17T09:08:08.583808+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:14:14.543065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583337+0000","last_change":"2024-09-17T09:07:56.254113+0000","last_active":"2024-09-17T09:08:08.583337+0000","last_peered":"2024-09-17T09:08:08.583337+0000","last_clean":"2024-09-17T09:08:08.583337+0000","last_became_active":"2024-09-17T09:07:56.253610+0000","last_became_peered":"2024-09-17T09:07:56.253610+0000","last_unstale":"2024-09-17T09:08:08.583337+0000","last_undegraded":"2024-09-17T09:08:08.583337+0000","last_fullsized":"2024-09-17T09:08:08.583337+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:51:00.410902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:02.704464+0000","last_change":"2024-09-17T09:07:58.274291+0000","last_active":"2024-09-17T09:09:02.704464+0000","last_peered":"2024-09-17T09:09:02.704464+0000","last_clean":"2024-09-17T09:09:02.704464+0000","last_became_active":"2024-09-17T09:07:58.273941+0000","last_became_peered":"2024-09-17T09:07:58.273941+0000","last_unstale":"2024-09-17T09:09:02.704464+0000","last_undegraded":"2024-09-17T09:09:02.704464+0000","last_fullsized":"2024-09-17T09:09:02.704464+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:48:56.584346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584451+0000","last_change":"2024-09-17T09:07:56.262979+0000","last_active":"2024-09-17T09:08:08.584451+0000","last_peered":"2024-09-17T09:08:08.584451+0000","last_clean":"2024-09-17T09:08:08.584451+0000","last_became_active":"2024-09-17T09:07:56.262580+0000","last_became_peered":"2024-09-17T09:07:56.262580+0000","last_unstale":"2024-09-17T09:08:08.584451+0000","last_undegraded":"2024-09-17T09:08:08.584451+0000","last_fullsized":"2024-09-17T09:08:08.584451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:20:31.479926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598345+0000","last_change":"2024-09-17T09:07:54.249565+0000","last_active":"2024-09-17T09:08:09.598345+0000","last_peered":"2024-09-17T09:08:09.598345+0000","last_clean":"2024-09-17T09:08:09.598345+0000","last_became_active":"2024-09-17T09:07:54.249322+0000","last_became_peered":"2024-09-17T09:07:54.249322+0000","last_unstale":"2024-09-17T09:08:09.598345+0000","last_undegraded":"2024-09-17T09:08:09.598345+0000","last_fullsized":"2024-09-17T09:08:09.598345+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:43.618527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:02.704459+0000","last_change":"2024-09-17T09:07:58.274124+0000","last_active":"2024-09-17T09:09:02.704459+0000","last_peered":"2024-09-17T09:09:02.704459+0000","last_clean":"2024-09-17T09:09:02.704459+0000","last_became_active":"2024-09-17T09:07:58.273476+0000","last_became_peered":"2024-09-17T09:07:58.273476+0000","last_unstale":"2024-09-17T09:09:02.704459+0000","last_undegraded":"2024-09-17T09:09:02.704459+0000","last_fullsized":"2024-09-17T09:09:02.704459+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:16.196820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582550+0000","last_change":"2024-09-17T09:08:00.277346+0000","last_active":"2024-09-17T09:08:08.582550+0000","last_peered":"2024-09-17T09:08:08.582550+0000","last_clean":"2024-09-17T09:08:08.582550+0000","last_became_active":"2024-09-17T09:08:00.277068+0000","last_became_peered":"2024-09-17T09:08:00.277068+0000","last_unstale":"2024-09-17T09:08:08.582550+0000","last_undegraded":"2024-09-17T09:08:08.582550+0000","last_fullsized":"2024-09-17T09:08:08.582550+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:03.417308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":49,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598072+0000","last_change":"2024-09-17T09:07:56.248310+0000","last_active":"2024-09-17T09:08:09.598072+0000","last_peered":"2024-09-17T09:08:09.598072+0000","last_clean":"2024-09-17T09:08:09.598072+0000","last_became_active":"2024-09-17T09:07:56.247999+0000","last_became_peered":"2024-09-17T09:07:56.247999+0000","last_unstale":"2024-09-17T09:08:09.598072+0000","last_undegraded":"2024-09-17T09:08:09.598072+0000","last_fullsized":"2024-09-17T09:08:09.598072+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:21.833823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598831+0000","last_change":"2024-09-17T09:07:54.248020+0000","last_active":"2024-09-17T09:08:09.598831+0000","last_peered":"2024-09-17T09:08:09.598831+0000","last_clean":"2024-09-17T09:08:09.598831+0000","last_became_active":"2024-09-17T09:07:54.247792+0000","last_became_peered":"2024-09-17T09:07:54.247792+0000","last_unstale":"2024-09-17T09:08:09.598831+0000","last_undegraded":"2024-09-17T09:08:09.598831+0000","last_fullsized":"2024-09-17T09:08:09.598831+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:08:58.091716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597615+0000","last_change":"2024-09-17T09:07:58.273144+0000","last_active":"2024-09-17T09:08:09.597615+0000","last_peered":"2024-09-17T09:08:09.597615+0000","last_clean":"2024-09-17T09:08:09.597615+0000","last_became_active":"2024-09-17T09:07:58.272850+0000","last_became_peered":"2024-09-17T09:07:58.272850+0000","last_unstale":"2024-09-17T09:08:09.597615+0000","last_undegraded":"2024-09-17T09:08:09.597615+0000","last_fullsized":"2024-09-17T09:08:09.597615+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:09.700309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597486+0000","last_change":"2024-09-17T09:08:00.269002+0000","last_active":"2024-09-17T09:08:09.597486+0000","last_peered":"2024-09-17T09:08:09.597486+0000","last_clean":"2024-09-17T09:08:09.597486+0000","last_became_active":"2024-09-17T09:08:00.268708+0000","last_became_peered":"2024-09-17T09:08:00.268708+0000","last_unstale":"2024-09-17T09:08:09.597486+0000","last_undegraded":"2024-09-17T09:08:09.597486+0000","last_fullsized":"2024-09-17T09:08:09.597486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:57.482275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":38,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596847+0000","last_change":"2024-09-17T09:07:56.250184+0000","last_active":"2024-09-17T09:08:09.596847+0000","last_peered":"2024-09-17T09:08:09.596847+0000","last_clean":"2024-09-17T09:08:09.596847+0000","last_became_active":"2024-09-17T09:07:56.249903+0000","last_became_peered":"2024-09-17T09:07:56.249903+0000","last_unstale":"2024-09-17T09:08:09.596847+0000","last_undegraded":"2024-09-17T09:08:09.596847+0000","last_fullsized":"2024-09-17T09:08:09.596847+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:59.727703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583615+0000","last_change":"2024-09-17T09:07:54.245488+0000","last_active":"2024-09-17T09:08:08.583615+0000","last_peered":"2024-09-17T09:08:08.583615+0000","last_clean":"2024-09-17T09:08:08.583615+0000","last_became_active":"2024-09-17T09:07:54.245095+0000","last_became_peered":"2024-09-17T09:07:54.245095+0000","last_unstale":"2024-09-17T09:08:08.583615+0000","last_undegraded":"2024-09-17T09:08:08.583615+0000","last_fullsized":"2024-09-17T09:08:08.583615+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:27.984879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:02.704412+0000","last_change":"2024-09-17T09:07:58.273778+0000","last_active":"2024-09-17T09:09:02.704412+0000","last_peered":"2024-09-17T09:09:02.704412+0000","last_clean":"2024-09-17T09:09:02.704412+0000","last_became_active":"2024-09-17T09:07:58.273431+0000","last_became_peered":"2024-09-17T09:07:58.273431+0000","last_unstale":"2024-09-17T09:09:02.704412+0000","last_undegraded":"2024-09-17T09:09:02.704412+0000","last_fullsized":"2024-09-17T09:09:02.704412+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:41:48.911678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596846+0000","last_change":"2024-09-17T09:08:00.411687+0000","last_active":"2024-09-17T09:08:09.596846+0000","last_peered":"2024-09-17T09:08:09.596846+0000","last_clean":"2024-09-17T09:08:09.596846+0000","last_became_active":"2024-09-17T09:08:00.411346+0000","last_became_peered":"2024-09-17T09:08:00.411346+0000","last_unstale":"2024-09-17T09:08:09.596846+0000","last_undegraded":"2024-09-17T09:08:09.596846+0000","last_fullsized":"2024-09-17T09:08:09.596846+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:48.090997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583101+0000","last_change":"2024-09-17T09:07:56.263388+0000","last_active":"2024-09-17T09:08:08.583101+0000","last_peered":"2024-09-17T09:08:08.583101+0000","last_clean":"2024-09-17T09:08:08.583101+0000","last_became_active":"2024-09-17T09:07:56.263032+0000","last_became_peered":"2024-09-17T09:07:56.263032+0000","last_unstale":"2024-09-17T09:08:08.583101+0000","last_undegraded":"2024-09-17T09:08:08.583101+0000","last_fullsized":"2024-09-17T09:08:08.583101+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:50:46.023755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582833+0000","last_change":"2024-09-17T09:07:54.250681+0000","last_active":"2024-09-17T09:08:08.582833+0000","last_peered":"2024-09-17T09:08:08.582833+0000","last_clean":"2024-09-17T09:08:08.582833+0000","last_became_active":"2024-09-17T09:07:54.250393+0000","last_became_peered":"2024-09-17T09:07:54.250393+0000","last_unstale":"2024-09-17T09:08:08.582833+0000","last_undegraded":"2024-09-17T09:08:08.582833+0000","last_fullsized":"2024-09-17T09:08:08.582833+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:27:11.495491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:02.704673+0000","last_change":"2024-09-17T09:07:58.280766+0000","last_active":"2024-09-17T09:09:02.704673+0000","last_peered":"2024-09-17T09:09:02.704673+0000","last_clean":"2024-09-17T09:09:02.704673+0000","last_became_active":"2024-09-17T09:07:58.280431+0000","last_became_peered":"2024-09-17T09:07:58.280431+0000","last_unstale":"2024-09-17T09:09:02.704673+0000","last_undegraded":"2024-09-17T09:09:02.704673+0000","last_fullsized":"2024-09-17T09:09:02.704673+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:43.136981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582666+0000","last_change":"2024-09-17T09:08:00.282626+0000","last_active":"2024-09-17T09:08:08.582666+0000","last_peered":"2024-09-17T09:08:08.582666+0000","last_clean":"2024-09-17T09:08:08.582666+0000","last_became_active":"2024-09-17T09:08:00.282326+0000","last_became_peered":"2024-09-17T09:08:00.282326+0000","last_unstale":"2024-09-17T09:08:08.582666+0000","last_undegraded":"2024-09-17T09:08:08.582666+0000","last_fullsized":"2024-09-17T09:08:08.582666+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:31:43.763578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":53,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584576+0000","last_change":"2024-09-17T09:07:56.252301+0000","last_active":"2024-09-17T09:08:08.584576+0000","last_peered":"2024-09-17T09:08:08.584576+0000","last_clean":"2024-09-17T09:08:08.584576+0000","last_became_active":"2024-09-17T09:07:56.252027+0000","last_became_peered":"2024-09-17T09:07:56.252027+0000","last_unstale":"2024-09-17T09:08:08.584576+0000","last_undegraded":"2024-09-17T09:08:08.584576+0000","last_fullsized":"2024-09-17T09:08:08.584576+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:10.437759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584305+0000","last_change":"2024-09-17T09:07:54.243714+0000","last_active":"2024-09-17T09:08:08.584305+0000","last_peered":"2024-09-17T09:08:08.584305+0000","last_clean":"2024-09-17T09:08:08.584305+0000","last_became_active":"2024-09-17T09:07:54.243433+0000","last_became_peered":"2024-09-17T09:07:54.243433+0000","last_unstale":"2024-09-17T09:08:08.584305+0000","last_undegraded":"2024-09-17T09:08:08.584305+0000","last_fullsized":"2024-09-17T09:08:08.584305+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:53.527157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547588+0000","last_change":"2024-09-17T09:07:58.276721+0000","last_active":"2024-09-17T09:08:09.547588+0000","last_peered":"2024-09-17T09:08:09.547588+0000","last_clean":"2024-09-17T09:08:09.547588+0000","last_became_active":"2024-09-17T09:07:58.276440+0000","last_became_peered":"2024-09-17T09:07:58.276440+0000","last_unstale":"2024-09-17T09:08:09.547588+0000","last_undegraded":"2024-09-17T09:08:09.547588+0000","last_fullsized":"2024-09-17T09:08:09.547588+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:47:12.201120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583566+0000","last_change":"2024-09-17T09:08:00.410764+0000","last_active":"2024-09-17T09:08:08.583566+0000","last_peered":"2024-09-17T09:08:08.583566+0000","last_clean":"2024-09-17T09:08:08.583566+0000","last_became_active":"2024-09-17T09:08:00.410377+0000","last_became_peered":"2024-09-17T09:08:00.410377+0000","last_unstale":"2024-09-17T09:08:08.583566+0000","last_undegraded":"2024-09-17T09:08:08.583566+0000","last_fullsized":"2024-09-17T09:08:08.583566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:50:20.460707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":80,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:02.704378+0000","last_change":"2024-09-17T09:07:56.261870+0000","last_active":"2024-09-17T09:09:02.704378+0000","last_peered":"2024-09-17T09:09:02.704378+0000","last_clean":"2024-09-17T09:09:02.704378+0000","last_became_active":"2024-09-17T09:07:56.261549+0000","last_became_peered":"2024-09-17T09:07:56.261549+0000","last_unstale":"2024-09-17T09:09:02.704378+0000","last_undegraded":"2024-09-17T09:09:02.704378+0000","last_fullsized":"2024-09-17T09:09:02.704378+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:11:08.133092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597518+0000","last_change":"2024-09-17T09:07:54.249306+0000","last_active":"2024-09-17T09:08:09.597518+0000","last_peered":"2024-09-17T09:08:09.597518+0000","last_clean":"2024-09-17T09:08:09.597518+0000","last_became_active":"2024-09-17T09:07:54.248862+0000","last_became_peered":"2024-09-17T09:07:54.248862+0000","last_unstale":"2024-09-17T09:08:09.597518+0000","last_undegraded":"2024-09-17T09:08:09.597518+0000","last_fullsized":"2024-09-17T09:08:09.597518+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:04.247970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":26,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582494+0000","last_change":"2024-09-17T09:07:58.272009+0000","last_active":"2024-09-17T09:08:08.582494+0000","last_peered":"2024-09-17T09:08:08.582494+0000","last_clean":"2024-09-17T09:08:08.582494+0000","last_became_active":"2024-09-17T09:07:54.248873+0000","last_became_peered":"2024-09-17T09:07:54.248873+0000","last_unstale":"2024-09-17T09:08:08.582494+0000","last_undegraded":"2024-09-17T09:08:08.582494+0000","last_fullsized":"2024-09-17T09:08:08.582494+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:04.719857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019774290000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583221+0000","last_change":"2024-09-17T09:07:58.278978+0000","last_active":"2024-09-17T09:08:08.583221+0000","last_peered":"2024-09-17T09:08:08.583221+0000","last_clean":"2024-09-17T09:08:08.583221+0000","last_became_active":"2024-09-17T09:07:58.278652+0000","last_became_peered":"2024-09-17T09:07:58.278652+0000","last_unstale":"2024-09-17T09:08:08.583221+0000","last_undegraded":"2024-09-17T09:08:08.583221+0000","last_fullsized":"2024-09-17T09:08:08.583221+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:05:23.314898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598350+0000","last_change":"2024-09-17T09:08:00.275542+0000","last_active":"2024-09-17T09:08:09.598350+0000","last_peered":"2024-09-17T09:08:09.598350+0000","last_clean":"2024-09-17T09:08:09.598350+0000","last_became_active":"2024-09-17T09:08:00.275267+0000","last_became_peered":"2024-09-17T09:08:00.275267+0000","last_unstale":"2024-09-17T09:08:09.598350+0000","last_undegraded":"2024-09-17T09:08:09.598350+0000","last_fullsized":"2024-09-17T09:08:09.598350+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:57.874829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":44,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584462+0000","last_change":"2024-09-17T09:07:56.262497+0000","last_active":"2024-09-17T09:08:08.584462+0000","last_peered":"2024-09-17T09:08:08.584462+0000","last_clean":"2024-09-17T09:08:08.584462+0000","last_became_active":"2024-09-17T09:07:56.262177+0000","last_became_peered":"2024-09-17T09:07:56.262177+0000","last_unstale":"2024-09-17T09:08:08.584462+0000","last_undegraded":"2024-09-17T09:08:08.584462+0000","last_fullsized":"2024-09-17T09:08:08.584462+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:22:31.229991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583435+0000","last_change":"2024-09-17T09:07:54.249798+0000","last_active":"2024-09-17T09:08:08.583435+0000","last_peered":"2024-09-17T09:08:08.583435+0000","last_clean":"2024-09-17T09:08:08.583435+0000","last_became_active":"2024-09-17T09:07:54.249520+0000","last_became_peered":"2024-09-17T09:07:54.249520+0000","last_unstale":"2024-09-17T09:08:08.583435+0000","last_undegraded":"2024-09-17T09:08:08.583435+0000","last_fullsized":"2024-09-17T09:08:08.583435+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:55.684631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":25,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597641+0000","last_change":"2024-09-17T09:07:58.263960+0000","last_active":"2024-09-17T09:08:09.597641+0000","last_peered":"2024-09-17T09:08:09.597641+0000","last_clean":"2024-09-17T09:08:09.597641+0000","last_became_active":"2024-09-17T09:07:54.250843+0000","last_became_peered":"2024-09-17T09:07:54.250843+0000","last_unstale":"2024-09-17T09:08:09.597641+0000","last_undegraded":"2024-09-17T09:08:09.597641+0000","last_fullsized":"2024-09-17T09:08:09.597641+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:24:22.827498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001848487,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597598+0000","last_change":"2024-09-17T09:07:58.273296+0000","last_active":"2024-09-17T09:08:09.597598+0000","last_peered":"2024-09-17T09:08:09.597598+0000","last_clean":"2024-09-17T09:08:09.597598+0000","last_became_active":"2024-09-17T09:07:58.273005+0000","last_became_peered":"2024-09-17T09:07:58.273005+0000","last_unstale":"2024-09-17T09:08:09.597598+0000","last_undegraded":"2024-09-17T09:08:09.597598+0000","last_fullsized":"2024-09-17T09:08:09.597598+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:16:29.679121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597800+0000","last_change":"2024-09-17T09:08:00.410953+0000","last_active":"2024-09-17T09:08:09.597800+0000","last_peered":"2024-09-17T09:08:09.597800+0000","last_clean":"2024-09-17T09:08:09.597800+0000","last_became_active":"2024-09-17T09:08:00.410647+0000","last_became_peered":"2024-09-17T09:08:00.410647+0000","last_unstale":"2024-09-17T09:08:09.597800+0000","last_undegraded":"2024-09-17T09:08:09.597800+0000","last_fullsized":"2024-09-17T09:08:09.597800+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:35.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":35,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583449+0000","last_change":"2024-09-17T09:07:56.257146+0000","last_active":"2024-09-17T09:08:08.583449+0000","last_peered":"2024-09-17T09:08:08.583449+0000","last_clean":"2024-09-17T09:08:08.583449+0000","last_became_active":"2024-09-17T09:07:56.256714+0000","last_became_peered":"2024-09-17T09:07:56.256714+0000","last_unstale":"2024-09-17T09:08:08.583449+0000","last_undegraded":"2024-09-17T09:08:08.583449+0000","last_fullsized":"2024-09-17T09:08:08.583449+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:49.062390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584148+0000","last_change":"2024-09-17T09:07:54.252996+0000","last_active":"2024-09-17T09:08:08.584148+0000","last_peered":"2024-09-17T09:08:08.584148+0000","last_clean":"2024-09-17T09:08:08.584148+0000","last_became_active":"2024-09-17T09:07:54.252722+0000","last_became_peered":"2024-09-17T09:07:54.252722+0000","last_unstale":"2024-09-17T09:08:08.584148+0000","last_undegraded":"2024-09-17T09:08:08.584148+0000","last_fullsized":"2024-09-17T09:08:08.584148+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:42:45.445678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":98,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:11.020552+0000","last_change":"2024-09-17T09:07:58.267104+0000","last_active":"2024-09-17T09:09:11.020552+0000","last_peered":"2024-09-17T09:09:11.020552+0000","last_clean":"2024-09-17T09:09:11.020552+0000","last_became_active":"2024-09-17T09:07:54.248032+0000","last_became_peered":"2024-09-17T09:07:54.248032+0000","last_unstale":"2024-09-17T09:09:11.020552+0000","last_undegraded":"2024-09-17T09:09:11.020552+0000","last_fullsized":"2024-09-17T09:09:11.020552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:09:30.903399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033981199999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583036+0000","last_change":"2024-09-17T09:07:58.278352+0000","last_active":"2024-09-17T09:08:08.583036+0000","last_peered":"2024-09-17T09:08:08.583036+0000","last_clean":"2024-09-17T09:08:08.583036+0000","last_became_active":"2024-09-17T09:07:58.278034+0000","last_became_peered":"2024-09-17T09:07:58.278034+0000","last_unstale":"2024-09-17T09:08:08.583036+0000","last_undegraded":"2024-09-17T09:08:08.583036+0000","last_fullsized":"2024-09-17T09:08:08.583036+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:43.714637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584001+0000","last_change":"2024-09-17T09:08:00.281783+0000","last_active":"2024-09-17T09:08:08.584001+0000","last_peered":"2024-09-17T09:08:08.584001+0000","last_clean":"2024-09-17T09:08:08.584001+0000","last_became_active":"2024-09-17T09:08:00.281462+0000","last_became_peered":"2024-09-17T09:08:00.281462+0000","last_unstale":"2024-09-17T09:08:08.584001+0000","last_undegraded":"2024-09-17T09:08:08.584001+0000","last_fullsized":"2024-09-17T09:08:08.584001+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:22.885975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"67'14","reported_seq":39,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547923+0000","last_change":"2024-09-17T09:07:56.250715+0000","last_active":"2024-09-17T09:08:09.547923+0000","last_peered":"2024-09-17T09:08:09.547923+0000","last_clean":"2024-09-17T09:08:09.547923+0000","last_became_active":"2024-09-17T09:07:56.250479+0000","last_became_peered":"2024-09-17T09:07:56.250479+0000","last_unstale":"2024-09-17T09:08:09.547923+0000","last_undegraded":"2024-09-17T09:08:09.547923+0000","last_fullsized":"2024-09-17T09:08:09.547923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:05.365570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598261+0000","last_change":"2024-09-17T09:07:54.251571+0000","last_active":"2024-09-17T09:08:09.598261+0000","last_peered":"2024-09-17T09:08:09.598261+0000","last_clean":"2024-09-17T09:08:09.598261+0000","last_became_active":"2024-09-17T09:07:54.251261+0000","last_became_peered":"2024-09-17T09:07:54.251261+0000","last_unstale":"2024-09-17T09:08:09.598261+0000","last_undegraded":"2024-09-17T09:08:09.598261+0000","last_fullsized":"2024-09-17T09:08:09.598261+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:02.409285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"69'128","reported_seq":265,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:08:33.416518+0000","last_change":"2024-09-17T09:07:44.430610+0000","last_active":"2024-09-17T09:08:33.416518+0000","last_peered":"2024-09-17T09:08:33.416518+0000","last_clean":"2024-09-17T09:08:33.416518+0000","last_became_active":"2024-09-17T09:07:44.430282+0000","last_became_peered":"2024-09-17T09:07:44.430282+0000","last_unstale":"2024-09-17T09:08:33.416518+0000","last_undegraded":"2024-09-17T09:08:33.416518+0000","last_fullsized":"2024-09-17T09:08:33.416518+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:05:43.688522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:05:43.688522+0000","last_clean_scrub_stamp":"2024-09-17T09:05:43.688522+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:38:07.261433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597465+0000","last_change":"2024-09-17T09:07:58.276429+0000","last_active":"2024-09-17T09:08:09.597465+0000","last_peered":"2024-09-17T09:08:09.597465+0000","last_clean":"2024-09-17T09:08:09.597465+0000","last_became_active":"2024-09-17T09:07:58.276143+0000","last_became_peered":"2024-09-17T09:07:58.276143+0000","last_unstale":"2024-09-17T09:08:09.597465+0000","last_undegraded":"2024-09-17T09:08:09.597465+0000","last_fullsized":"2024-09-17T09:08:09.597465+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:56:47.012931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582956+0000","last_change":"2024-09-17T09:08:00.282321+0000","last_active":"2024-09-17T09:08:08.582956+0000","last_peered":"2024-09-17T09:08:08.582956+0000","last_clean":"2024-09-17T09:08:08.582956+0000","last_became_active":"2024-09-17T09:08:00.282042+0000","last_became_peered":"2024-09-17T09:08:00.282042+0000","last_unstale":"2024-09-17T09:08:08.582956+0000","last_undegraded":"2024-09-17T09:08:08.582956+0000","last_fullsized":"2024-09-17T09:08:08.582956+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:13:06.700284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597591+0000","last_change":"2024-09-17T09:07:56.261402+0000","last_active":"2024-09-17T09:08:09.597591+0000","last_peered":"2024-09-17T09:08:09.597591+0000","last_clean":"2024-09-17T09:08:09.597591+0000","last_became_active":"2024-09-17T09:07:56.261120+0000","last_became_peered":"2024-09-17T09:07:56.261120+0000","last_unstale":"2024-09-17T09:08:09.597591+0000","last_undegraded":"2024-09-17T09:08:09.597591+0000","last_fullsized":"2024-09-17T09:08:09.597591+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:44.704514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597162+0000","last_change":"2024-09-17T09:07:54.247019+0000","last_active":"2024-09-17T09:08:09.597162+0000","last_peered":"2024-09-17T09:08:09.597162+0000","last_clean":"2024-09-17T09:08:09.597162+0000","last_became_active":"2024-09-17T09:07:54.246556+0000","last_became_peered":"2024-09-17T09:07:54.246556+0000","last_unstale":"2024-09-17T09:08:09.597162+0000","last_undegraded":"2024-09-17T09:08:09.597162+0000","last_fullsized":"2024-09-17T09:08:09.597162+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:39:10.724050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582885+0000","last_change":"2024-09-17T09:07:58.279814+0000","last_active":"2024-09-17T09:08:08.582885+0000","last_peered":"2024-09-17T09:08:08.582885+0000","last_clean":"2024-09-17T09:08:08.582885+0000","last_became_active":"2024-09-17T09:07:58.279498+0000","last_became_peered":"2024-09-17T09:07:58.279498+0000","last_unstale":"2024-09-17T09:08:08.582885+0000","last_undegraded":"2024-09-17T09:08:08.582885+0000","last_fullsized":"2024-09-17T09:08:08.582885+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:08:32.224061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597900+0000","last_change":"2024-09-17T09:08:00.411235+0000","last_active":"2024-09-17T09:08:09.597900+0000","last_peered":"2024-09-17T09:08:09.597900+0000","last_clean":"2024-09-17T09:08:09.597900+0000","last_became_active":"2024-09-17T09:08:00.410915+0000","last_became_peered":"2024-09-17T09:08:00.410915+0000","last_unstale":"2024-09-17T09:08:09.597900+0000","last_undegraded":"2024-09-17T09:08:09.597900+0000","last_fullsized":"2024-09-17T09:08:09.597900+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:57.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":45,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597624+0000","last_change":"2024-09-17T09:07:56.263357+0000","last_active":"2024-09-17T09:08:09.597624+0000","last_peered":"2024-09-17T09:08:09.597624+0000","last_clean":"2024-09-17T09:08:09.597624+0000","last_became_active":"2024-09-17T09:07:56.263029+0000","last_became_peered":"2024-09-17T09:07:56.263029+0000","last_unstale":"2024-09-17T09:08:09.597624+0000","last_undegraded":"2024-09-17T09:08:09.597624+0000","last_fullsized":"2024-09-17T09:08:09.597624+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.854310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547739+0000","last_change":"2024-09-17T09:07:54.251577+0000","last_active":"2024-09-17T09:08:09.547739+0000","last_peered":"2024-09-17T09:08:09.547739+0000","last_clean":"2024-09-17T09:08:09.547739+0000","last_became_active":"2024-09-17T09:07:54.251335+0000","last_became_peered":"2024-09-17T09:07:54.251335+0000","last_unstale":"2024-09-17T09:08:09.547739+0000","last_undegraded":"2024-09-17T09:08:09.547739+0000","last_fullsized":"2024-09-17T09:08:09.547739+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:31.606953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584078+0000","last_change":"2024-09-17T09:07:58.278665+0000","last_active":"2024-09-17T09:08:08.584078+0000","last_peered":"2024-09-17T09:08:08.584078+0000","last_clean":"2024-09-17T09:08:08.584078+0000","last_became_active":"2024-09-17T09:07:58.278335+0000","last_became_peered":"2024-09-17T09:07:58.278335+0000","last_unstale":"2024-09-17T09:08:08.584078+0000","last_undegraded":"2024-09-17T09:08:08.584078+0000","last_fullsized":"2024-09-17T09:08:08.584078+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:30.555707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597203+0000","last_change":"2024-09-17T09:08:00.271281+0000","last_active":"2024-09-17T09:08:09.597203+0000","last_peered":"2024-09-17T09:08:09.597203+0000","last_clean":"2024-09-17T09:08:09.597203+0000","last_became_active":"2024-09-17T09:08:00.270928+0000","last_became_peered":"2024-09-17T09:08:00.270928+0000","last_unstale":"2024-09-17T09:08:09.597203+0000","last_undegraded":"2024-09-17T09:08:09.597203+0000","last_fullsized":"2024-09-17T09:08:09.597203+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:24:12.214954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597049+0000","last_change":"2024-09-17T09:07:56.255100+0000","last_active":"2024-09-17T09:08:09.597049+0000","last_peered":"2024-09-17T09:08:09.597049+0000","last_clean":"2024-09-17T09:08:09.597049+0000","last_became_active":"2024-09-17T09:07:56.254843+0000","last_became_peered":"2024-09-17T09:07:56.254843+0000","last_unstale":"2024-09-17T09:08:09.597049+0000","last_undegraded":"2024-09-17T09:08:09.597049+0000","last_fullsized":"2024-09-17T09:08:09.597049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:10.512387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598558+0000","last_change":"2024-09-17T09:07:54.249030+0000","last_active":"2024-09-17T09:08:09.598558+0000","last_peered":"2024-09-17T09:08:09.598558+0000","last_clean":"2024-09-17T09:08:09.598558+0000","last_became_active":"2024-09-17T09:07:54.248698+0000","last_became_peered":"2024-09-17T09:07:54.248698+0000","last_unstale":"2024-09-17T09:08:09.598558+0000","last_undegraded":"2024-09-17T09:08:09.598558+0000","last_fullsized":"2024-09-17T09:08:09.598558+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:56:07.518482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:02.704385+0000","last_change":"2024-09-17T09:07:58.273256+0000","last_active":"2024-09-17T09:09:02.704385+0000","last_peered":"2024-09-17T09:09:02.704385+0000","last_clean":"2024-09-17T09:09:02.704385+0000","last_became_active":"2024-09-17T09:07:58.272951+0000","last_became_peered":"2024-09-17T09:07:58.272951+0000","last_unstale":"2024-09-17T09:09:02.704385+0000","last_undegraded":"2024-09-17T09:09:02.704385+0000","last_fullsized":"2024-09-17T09:09:02.704385+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:26:06.792961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596973+0000","last_change":"2024-09-17T09:08:00.278487+0000","last_active":"2024-09-17T09:08:09.596973+0000","last_peered":"2024-09-17T09:08:09.596973+0000","last_clean":"2024-09-17T09:08:09.596973+0000","last_became_active":"2024-09-17T09:08:00.278151+0000","last_became_peered":"2024-09-17T09:08:00.278151+0000","last_unstale":"2024-09-17T09:08:09.596973+0000","last_undegraded":"2024-09-17T09:08:09.596973+0000","last_fullsized":"2024-09-17T09:08:09.596973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:04.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584484+0000","last_change":"2024-09-17T09:07:56.259436+0000","last_active":"2024-09-17T09:08:08.584484+0000","last_peered":"2024-09-17T09:08:08.584484+0000","last_clean":"2024-09-17T09:08:08.584484+0000","last_became_active":"2024-09-17T09:07:56.258702+0000","last_became_peered":"2024-09-17T09:07:56.258702+0000","last_unstale":"2024-09-17T09:08:08.584484+0000","last_undegraded":"2024-09-17T09:08:08.584484+0000","last_fullsized":"2024-09-17T09:08:08.584484+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:31.375700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582474+0000","last_change":"2024-09-17T09:07:54.254581+0000","last_active":"2024-09-17T09:08:08.582474+0000","last_peered":"2024-09-17T09:08:08.582474+0000","last_clean":"2024-09-17T09:08:08.582474+0000","last_became_active":"2024-09-17T09:07:54.254270+0000","last_became_peered":"2024-09-17T09:07:54.254270+0000","last_unstale":"2024-09-17T09:08:08.582474+0000","last_undegraded":"2024-09-17T09:08:08.582474+0000","last_fullsized":"2024-09-17T09:08:08.582474+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:22.602874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597062+0000","last_change":"2024-09-17T09:07:58.269859+0000","last_active":"2024-09-17T09:08:09.597062+0000","last_peered":"2024-09-17T09:08:09.597062+0000","last_clean":"2024-09-17T09:08:09.597062+0000","last_became_active":"2024-09-17T09:07:58.269553+0000","last_became_peered":"2024-09-17T09:07:58.269553+0000","last_unstale":"2024-09-17T09:08:09.597062+0000","last_undegraded":"2024-09-17T09:08:09.597062+0000","last_fullsized":"2024-09-17T09:08:09.597062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:23:12.227060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584074+0000","last_change":"2024-09-17T09:08:00.283656+0000","last_active":"2024-09-17T09:08:08.584074+0000","last_peered":"2024-09-17T09:08:08.584074+0000","last_clean":"2024-09-17T09:08:08.584074+0000","last_became_active":"2024-09-17T09:08:00.283326+0000","last_became_peered":"2024-09-17T09:08:00.283326+0000","last_unstale":"2024-09-17T09:08:08.584074+0000","last_undegraded":"2024-09-17T09:08:08.584074+0000","last_fullsized":"2024-09-17T09:08:08.584074+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:10.953066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":48,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547636+0000","last_change":"2024-09-17T09:07:56.256656+0000","last_active":"2024-09-17T09:08:09.547636+0000","last_peered":"2024-09-17T09:08:09.547636+0000","last_clean":"2024-09-17T09:08:09.547636+0000","last_became_active":"2024-09-17T09:07:56.256381+0000","last_became_peered":"2024-09-17T09:07:56.256381+0000","last_unstale":"2024-09-17T09:08:09.547636+0000","last_undegraded":"2024-09-17T09:08:09.547636+0000","last_fullsized":"2024-09-17T09:08:09.547636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:28:28.393034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598035+0000","last_change":"2024-09-17T09:07:54.253742+0000","last_active":"2024-09-17T09:08:09.598035+0000","last_peered":"2024-09-17T09:08:09.598035+0000","last_clean":"2024-09-17T09:08:09.598035+0000","last_became_active":"2024-09-17T09:07:54.253250+0000","last_became_peered":"2024-09-17T09:07:54.253250+0000","last_unstale":"2024-09-17T09:08:09.598035+0000","last_undegraded":"2024-09-17T09:08:09.598035+0000","last_fullsized":"2024-09-17T09:08:09.598035+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:16.198331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597059+0000","last_change":"2024-09-17T09:07:58.276592+0000","last_active":"2024-09-17T09:08:09.597059+0000","last_peered":"2024-09-17T09:08:09.597059+0000","last_clean":"2024-09-17T09:08:09.597059+0000","last_became_active":"2024-09-17T09:07:58.276316+0000","last_became_peered":"2024-09-17T09:07:58.276316+0000","last_unstale":"2024-09-17T09:08:09.597059+0000","last_undegraded":"2024-09-17T09:08:09.597059+0000","last_fullsized":"2024-09-17T09:08:09.597059+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:47.085654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596932+0000","last_change":"2024-09-17T09:08:00.284383+0000","last_active":"2024-09-17T09:08:09.596932+0000","last_peered":"2024-09-17T09:08:09.596932+0000","last_clean":"2024-09-17T09:08:09.596932+0000","last_became_active":"2024-09-17T09:08:00.284154+0000","last_became_peered":"2024-09-17T09:08:00.284154+0000","last_unstale":"2024-09-17T09:08:09.596932+0000","last_undegraded":"2024-09-17T09:08:09.596932+0000","last_fullsized":"2024-09-17T09:08:09.596932+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:17.442555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":40,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597030+0000","last_change":"2024-09-17T09:07:56.262185+0000","last_active":"2024-09-17T09:08:09.597030+0000","last_peered":"2024-09-17T09:08:09.597030+0000","last_clean":"2024-09-17T09:08:09.597030+0000","last_became_active":"2024-09-17T09:07:56.261514+0000","last_became_peered":"2024-09-17T09:07:56.261514+0000","last_unstale":"2024-09-17T09:08:09.597030+0000","last_undegraded":"2024-09-17T09:08:09.597030+0000","last_fullsized":"2024-09-17T09:08:09.597030+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:00:07.002282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598160+0000","last_change":"2024-09-17T09:07:54.251430+0000","last_active":"2024-09-17T09:08:09.598160+0000","last_peered":"2024-09-17T09:08:09.598160+0000","last_clean":"2024-09-17T09:08:09.598160+0000","last_became_active":"2024-09-17T09:07:54.251145+0000","last_became_peered":"2024-09-17T09:07:54.251145+0000","last_unstale":"2024-09-17T09:08:09.598160+0000","last_undegraded":"2024-09-17T09:08:09.598160+0000","last_fullsized":"2024-09-17T09:08:09.598160+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:01.036138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596983+0000","last_change":"2024-09-17T09:07:58.272732+0000","last_active":"2024-09-17T09:08:09.596983+0000","last_peered":"2024-09-17T09:08:09.596983+0000","last_clean":"2024-09-17T09:08:09.596983+0000","last_became_active":"2024-09-17T09:07:58.272334+0000","last_became_peered":"2024-09-17T09:07:58.272334+0000","last_unstale":"2024-09-17T09:08:09.596983+0000","last_undegraded":"2024-09-17T09:08:09.596983+0000","last_fullsized":"2024-09-17T09:08:09.596983+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:55:17.168854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597782+0000","last_change":"2024-09-17T09:08:00.279405+0000","last_active":"2024-09-17T09:08:09.597782+0000","last_peered":"2024-09-17T09:08:09.597782+0000","last_clean":"2024-09-17T09:08:09.597782+0000","last_became_active":"2024-09-17T09:08:00.279057+0000","last_became_peered":"2024-09-17T09:08:00.279057+0000","last_unstale":"2024-09-17T09:08:09.597782+0000","last_undegraded":"2024-09-17T09:08:09.597782+0000","last_fullsized":"2024-09-17T09:08:09.597782+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:51.833526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582074+0000","last_change":"2024-09-17T09:07:56.249475+0000","last_active":"2024-09-17T09:08:08.582074+0000","last_peered":"2024-09-17T09:08:08.582074+0000","last_clean":"2024-09-17T09:08:08.582074+0000","last_became_active":"2024-09-17T09:07:56.248230+0000","last_became_peered":"2024-09-17T09:07:56.248230+0000","last_unstale":"2024-09-17T09:08:08.582074+0000","last_undegraded":"2024-09-17T09:08:08.582074+0000","last_fullsized":"2024-09-17T09:08:08.582074+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:43.442867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":26,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597854+0000","last_change":"2024-09-17T09:07:54.252748+0000","last_active":"2024-09-17T09:08:09.597854+0000","last_peered":"2024-09-17T09:08:09.597854+0000","last_clean":"2024-09-17T09:08:09.597854+0000","last_became_active":"2024-09-17T09:07:54.249944+0000","last_became_peered":"2024-09-17T09:07:54.249944+0000","last_unstale":"2024-09-17T09:08:09.597854+0000","last_undegraded":"2024-09-17T09:08:09.597854+0000","last_fullsized":"2024-09-17T09:08:09.597854+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:16.653591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:02.704466+0000","last_change":"2024-09-17T09:07:58.274512+0000","last_active":"2024-09-17T09:09:02.704466+0000","last_peered":"2024-09-17T09:09:02.704466+0000","last_clean":"2024-09-17T09:09:02.704466+0000","last_became_active":"2024-09-17T09:07:58.274254+0000","last_became_peered":"2024-09-17T09:07:58.274254+0000","last_unstale":"2024-09-17T09:09:02.704466+0000","last_undegraded":"2024-09-17T09:09:02.704466+0000","last_fullsized":"2024-09-17T09:09:02.704466+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:15:32.353987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582242+0000","last_change":"2024-09-17T09:08:00.410948+0000","last_active":"2024-09-17T09:08:08.582242+0000","last_peered":"2024-09-17T09:08:08.582242+0000","last_clean":"2024-09-17T09:08:08.582242+0000","last_became_active":"2024-09-17T09:08:00.410659+0000","last_became_peered":"2024-09-17T09:08:00.410659+0000","last_unstale":"2024-09-17T09:08:08.582242+0000","last_undegraded":"2024-09-17T09:08:08.582242+0000","last_fullsized":"2024-09-17T09:08:08.582242+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:15.009371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547550+0000","last_change":"2024-09-17T09:07:54.249989+0000","last_active":"2024-09-17T09:08:09.547550+0000","last_peered":"2024-09-17T09:08:09.547550+0000","last_clean":"2024-09-17T09:08:09.547550+0000","last_became_active":"2024-09-17T09:07:54.249717+0000","last_became_peered":"2024-09-17T09:07:54.249717+0000","last_unstale":"2024-09-17T09:08:09.547550+0000","last_undegraded":"2024-09-17T09:08:09.547550+0000","last_fullsized":"2024-09-17T09:08:09.547550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:58:26.320619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":26,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598770+0000","last_change":"2024-09-17T09:07:56.259126+0000","last_active":"2024-09-17T09:08:09.598770+0000","last_peered":"2024-09-17T09:08:09.598770+0000","last_clean":"2024-09-17T09:08:09.598770+0000","last_became_active":"2024-09-17T09:07:56.258897+0000","last_became_peered":"2024-09-17T09:07:56.258897+0000","last_unstale":"2024-09-17T09:08:09.598770+0000","last_undegraded":"2024-09-17T09:08:09.598770+0000","last_fullsized":"2024-09-17T09:08:09.598770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:39.212779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582799+0000","last_change":"2024-09-17T09:07:58.272347+0000","last_active":"2024-09-17T09:08:08.582799+0000","last_peered":"2024-09-17T09:08:08.582799+0000","last_clean":"2024-09-17T09:08:08.582799+0000","last_became_active":"2024-09-17T09:07:58.272031+0000","last_became_peered":"2024-09-17T09:07:58.272031+0000","last_unstale":"2024-09-17T09:08:08.582799+0000","last_undegraded":"2024-09-17T09:08:08.582799+0000","last_fullsized":"2024-09-17T09:08:08.582799+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:15.727684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597722+0000","last_change":"2024-09-17T09:08:00.412017+0000","last_active":"2024-09-17T09:08:09.597722+0000","last_peered":"2024-09-17T09:08:09.597722+0000","last_clean":"2024-09-17T09:08:09.597722+0000","last_became_active":"2024-09-17T09:08:00.411721+0000","last_became_peered":"2024-09-17T09:08:00.411721+0000","last_unstale":"2024-09-17T09:08:09.597722+0000","last_undegraded":"2024-09-17T09:08:09.597722+0000","last_fullsized":"2024-09-17T09:08:09.597722+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:21.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583536+0000","last_change":"2024-09-17T09:07:56.256979+0000","last_active":"2024-09-17T09:08:08.583536+0000","last_peered":"2024-09-17T09:08:08.583536+0000","last_clean":"2024-09-17T09:08:08.583536+0000","last_became_active":"2024-09-17T09:07:56.256324+0000","last_became_peered":"2024-09-17T09:07:56.256324+0000","last_unstale":"2024-09-17T09:08:08.583536+0000","last_undegraded":"2024-09-17T09:08:08.583536+0000","last_fullsized":"2024-09-17T09:08:08.583536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:53.378196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597429+0000","last_change":"2024-09-17T09:07:54.250742+0000","last_active":"2024-09-17T09:08:09.597429+0000","last_peered":"2024-09-17T09:08:09.597429+0000","last_clean":"2024-09-17T09:08:09.597429+0000","last_became_active":"2024-09-17T09:07:54.248384+0000","last_became_peered":"2024-09-17T09:07:54.248384+0000","last_unstale":"2024-09-17T09:08:09.597429+0000","last_undegraded":"2024-09-17T09:08:09.597429+0000","last_fullsized":"2024-09-17T09:08:09.597429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:16.872233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598688+0000","last_change":"2024-09-17T09:07:58.275761+0000","last_active":"2024-09-17T09:08:09.598688+0000","last_peered":"2024-09-17T09:08:09.598688+0000","last_clean":"2024-09-17T09:08:09.598688+0000","last_became_active":"2024-09-17T09:07:58.275481+0000","last_became_peered":"2024-09-17T09:07:58.275481+0000","last_unstale":"2024-09-17T09:08:09.598688+0000","last_undegraded":"2024-09-17T09:08:09.598688+0000","last_fullsized":"2024-09-17T09:08:09.598688+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:14.444429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597295+0000","last_change":"2024-09-17T09:08:00.275968+0000","last_active":"2024-09-17T09:08:09.597295+0000","last_peered":"2024-09-17T09:08:09.597295+0000","last_clean":"2024-09-17T09:08:09.597295+0000","last_became_active":"2024-09-17T09:08:00.275656+0000","last_became_peered":"2024-09-17T09:08:00.275656+0000","last_unstale":"2024-09-17T09:08:09.597295+0000","last_undegraded":"2024-09-17T09:08:09.597295+0000","last_fullsized":"2024-09-17T09:08:09.597295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:49:51.821355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582710+0000","last_change":"2024-09-17T09:07:56.254319+0000","last_active":"2024-09-17T09:08:08.582710+0000","last_peered":"2024-09-17T09:08:08.582710+0000","last_clean":"2024-09-17T09:08:08.582710+0000","last_became_active":"2024-09-17T09:07:56.254008+0000","last_became_peered":"2024-09-17T09:07:56.254008+0000","last_unstale":"2024-09-17T09:08:08.582710+0000","last_undegraded":"2024-09-17T09:08:08.582710+0000","last_fullsized":"2024-09-17T09:08:08.582710+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:45.719498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583909+0000","last_change":"2024-09-17T09:07:54.250064+0000","last_active":"2024-09-17T09:08:08.583909+0000","last_peered":"2024-09-17T09:08:08.583909+0000","last_clean":"2024-09-17T09:08:08.583909+0000","last_became_active":"2024-09-17T09:07:54.249746+0000","last_became_peered":"2024-09-17T09:07:54.249746+0000","last_unstale":"2024-09-17T09:08:08.583909+0000","last_undegraded":"2024-09-17T09:08:08.583909+0000","last_fullsized":"2024-09-17T09:08:08.583909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:00:19.112070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:02.704483+0000","last_change":"2024-09-17T09:07:58.272858+0000","last_active":"2024-09-17T09:09:02.704483+0000","last_peered":"2024-09-17T09:09:02.704483+0000","last_clean":"2024-09-17T09:09:02.704483+0000","last_became_active":"2024-09-17T09:07:58.272525+0000","last_became_peered":"2024-09-17T09:07:58.272525+0000","last_unstale":"2024-09-17T09:09:02.704483+0000","last_undegraded":"2024-09-17T09:09:02.704483+0000","last_fullsized":"2024-09-17T09:09:02.704483+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:24:12.623502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597860+0000","last_change":"2024-09-17T09:08:00.273280+0000","last_active":"2024-09-17T09:08:09.597860+0000","last_peered":"2024-09-17T09:08:09.597860+0000","last_clean":"2024-09-17T09:08:09.597860+0000","last_became_active":"2024-09-17T09:08:00.272978+0000","last_became_peered":"2024-09-17T09:08:00.272978+0000","last_unstale":"2024-09-17T09:08:09.597860+0000","last_undegraded":"2024-09-17T09:08:09.597860+0000","last_fullsized":"2024-09-17T09:08:09.597860+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:16.164898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598634+0000","last_change":"2024-09-17T09:07:56.257915+0000","last_active":"2024-09-17T09:08:09.598634+0000","last_peered":"2024-09-17T09:08:09.598634+0000","last_clean":"2024-09-17T09:08:09.598634+0000","last_became_active":"2024-09-17T09:07:56.257657+0000","last_became_peered":"2024-09-17T09:07:56.257657+0000","last_unstale":"2024-09-17T09:08:09.598634+0000","last_undegraded":"2024-09-17T09:08:09.598634+0000","last_fullsized":"2024-09-17T09:08:09.598634+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:49.233892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597612+0000","last_change":"2024-09-17T09:07:54.252390+0000","last_active":"2024-09-17T09:08:09.597612+0000","last_peered":"2024-09-17T09:08:09.597612+0000","last_clean":"2024-09-17T09:08:09.597612+0000","last_became_active":"2024-09-17T09:07:54.251943+0000","last_became_peered":"2024-09-17T09:07:54.251943+0000","last_unstale":"2024-09-17T09:08:09.597612+0000","last_undegraded":"2024-09-17T09:08:09.597612+0000","last_fullsized":"2024-09-17T09:08:09.597612+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:10.164114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:02.704427+0000","last_change":"2024-09-17T09:07:58.278470+0000","last_active":"2024-09-17T09:09:02.704427+0000","last_peered":"2024-09-17T09:09:02.704427+0000","last_clean":"2024-09-17T09:09:02.704427+0000","last_became_active":"2024-09-17T09:07:58.278208+0000","last_became_peered":"2024-09-17T09:07:58.278208+0000","last_unstale":"2024-09-17T09:09:02.704427+0000","last_undegraded":"2024-09-17T09:09:02.704427+0000","last_fullsized":"2024-09-17T09:09:02.704427+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:12.960474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583079+0000","last_change":"2024-09-17T09:08:00.282545+0000","last_active":"2024-09-17T09:08:08.583079+0000","last_peered":"2024-09-17T09:08:08.583079+0000","last_clean":"2024-09-17T09:08:08.583079+0000","last_became_active":"2024-09-17T09:08:00.282264+0000","last_became_peered":"2024-09-17T09:08:00.282264+0000","last_unstale":"2024-09-17T09:08:08.583079+0000","last_undegraded":"2024-09-17T09:08:08.583079+0000","last_fullsized":"2024-09-17T09:08:08.583079+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:15.713272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597083+0000","last_change":"2024-09-17T09:07:56.261507+0000","last_active":"2024-09-17T09:08:09.597083+0000","last_peered":"2024-09-17T09:08:09.597083+0000","last_clean":"2024-09-17T09:08:09.597083+0000","last_became_active":"2024-09-17T09:07:56.261188+0000","last_became_peered":"2024-09-17T09:07:56.261188+0000","last_unstale":"2024-09-17T09:08:09.597083+0000","last_undegraded":"2024-09-17T09:08:09.597083+0000","last_fullsized":"2024-09-17T09:08:09.597083+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:59.493678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597800+0000","last_change":"2024-09-17T09:07:54.251442+0000","last_active":"2024-09-17T09:08:09.597800+0000","last_peered":"2024-09-17T09:08:09.597800+0000","last_clean":"2024-09-17T09:08:09.597800+0000","last_became_active":"2024-09-17T09:07:54.251172+0000","last_became_peered":"2024-09-17T09:07:54.251172+0000","last_unstale":"2024-09-17T09:08:09.597800+0000","last_undegraded":"2024-09-17T09:08:09.597800+0000","last_fullsized":"2024-09-17T09:08:09.597800+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:31.284871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582936+0000","last_change":"2024-09-17T09:07:58.270640+0000","last_active":"2024-09-17T09:08:08.582936+0000","last_peered":"2024-09-17T09:08:08.582936+0000","last_clean":"2024-09-17T09:08:08.582936+0000","last_became_active":"2024-09-17T09:07:58.270338+0000","last_became_peered":"2024-09-17T09:07:58.270338+0000","last_unstale":"2024-09-17T09:08:08.582936+0000","last_undegraded":"2024-09-17T09:08:08.582936+0000","last_fullsized":"2024-09-17T09:08:08.582936+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:28:10.388495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597220+0000","last_change":"2024-09-17T09:08:00.278406+0000","last_active":"2024-09-17T09:08:09.597220+0000","last_peered":"2024-09-17T09:08:09.597220+0000","last_clean":"2024-09-17T09:08:09.597220+0000","last_became_active":"2024-09-17T09:08:00.278093+0000","last_became_peered":"2024-09-17T09:08:00.278093+0000","last_unstale":"2024-09-17T09:08:09.597220+0000","last_undegraded":"2024-09-17T09:08:09.597220+0000","last_fullsized":"2024-09-17T09:08:09.597220+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:38.162549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583201+0000","last_change":"2024-09-17T09:07:56.263088+0000","last_active":"2024-09-17T09:08:08.583201+0000","last_peered":"2024-09-17T09:08:08.583201+0000","last_clean":"2024-09-17T09:08:08.583201+0000","last_became_active":"2024-09-17T09:07:56.262820+0000","last_became_peered":"2024-09-17T09:07:56.262820+0000","last_unstale":"2024-09-17T09:08:08.583201+0000","last_undegraded":"2024-09-17T09:08:08.583201+0000","last_fullsized":"2024-09-17T09:08:08.583201+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:15:45.983694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597921+0000","last_change":"2024-09-17T09:07:54.251865+0000","last_active":"2024-09-17T09:08:09.597921+0000","last_peered":"2024-09-17T09:08:09.597921+0000","last_clean":"2024-09-17T09:08:09.597921+0000","last_became_active":"2024-09-17T09:07:54.251544+0000","last_became_peered":"2024-09-17T09:07:54.251544+0000","last_unstale":"2024-09-17T09:08:09.597921+0000","last_undegraded":"2024-09-17T09:08:09.597921+0000","last_fullsized":"2024-09-17T09:08:09.597921+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:34.212625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597877+0000","last_change":"2024-09-17T09:07:58.277185+0000","last_active":"2024-09-17T09:08:09.597877+0000","last_peered":"2024-09-17T09:08:09.597877+0000","last_clean":"2024-09-17T09:08:09.597877+0000","last_became_active":"2024-09-17T09:07:58.276904+0000","last_became_peered":"2024-09-17T09:07:58.276904+0000","last_unstale":"2024-09-17T09:08:09.597877+0000","last_undegraded":"2024-09-17T09:08:09.597877+0000","last_fullsized":"2024-09-17T09:08:09.597877+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:46.991179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582480+0000","last_change":"2024-09-17T09:08:00.280989+0000","last_active":"2024-09-17T09:08:08.582480+0000","last_peered":"2024-09-17T09:08:08.582480+0000","last_clean":"2024-09-17T09:08:08.582480+0000","last_became_active":"2024-09-17T09:08:00.280680+0000","last_became_peered":"2024-09-17T09:08:00.280680+0000","last_unstale":"2024-09-17T09:08:08.582480+0000","last_undegraded":"2024-09-17T09:08:08.582480+0000","last_fullsized":"2024-09-17T09:08:08.582480+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:13:28.443695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597365+0000","last_change":"2024-09-17T09:07:56.247428+0000","last_active":"2024-09-17T09:08:09.597365+0000","last_peered":"2024-09-17T09:08:09.597365+0000","last_clean":"2024-09-17T09:08:09.597365+0000","last_became_active":"2024-09-17T09:07:56.247086+0000","last_became_peered":"2024-09-17T09:07:56.247086+0000","last_unstale":"2024-09-17T09:08:09.597365+0000","last_undegraded":"2024-09-17T09:08:09.597365+0000","last_fullsized":"2024-09-17T09:08:09.597365+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:10.997594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582807+0000","last_change":"2024-09-17T09:07:54.246150+0000","last_active":"2024-09-17T09:08:08.582807+0000","last_peered":"2024-09-17T09:08:08.582807+0000","last_clean":"2024-09-17T09:08:08.582807+0000","last_became_active":"2024-09-17T09:07:54.245853+0000","last_became_peered":"2024-09-17T09:07:54.245853+0000","last_unstale":"2024-09-17T09:08:08.582807+0000","last_undegraded":"2024-09-17T09:08:08.582807+0000","last_fullsized":"2024-09-17T09:08:08.582807+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:43.443395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596638+0000","last_change":"2024-09-17T09:07:58.275230+0000","last_active":"2024-09-17T09:08:09.596638+0000","last_peered":"2024-09-17T09:08:09.596638+0000","last_clean":"2024-09-17T09:08:09.596638+0000","last_became_active":"2024-09-17T09:07:58.274829+0000","last_became_peered":"2024-09-17T09:07:58.274829+0000","last_unstale":"2024-09-17T09:08:09.596638+0000","last_undegraded":"2024-09-17T09:08:09.596638+0000","last_fullsized":"2024-09-17T09:08:09.596638+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:43.808216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598059+0000","last_change":"2024-09-17T09:08:00.413063+0000","last_active":"2024-09-17T09:08:09.598059+0000","last_peered":"2024-09-17T09:08:09.598059+0000","last_clean":"2024-09-17T09:08:09.598059+0000","last_became_active":"2024-09-17T09:08:00.412540+0000","last_became_peered":"2024-09-17T09:08:00.412540+0000","last_unstale":"2024-09-17T09:08:09.598059+0000","last_undegraded":"2024-09-17T09:08:09.598059+0000","last_fullsized":"2024-09-17T09:08:09.598059+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:42.269188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598072+0000","last_change":"2024-09-17T09:07:56.259721+0000","last_active":"2024-09-17T09:08:09.598072+0000","last_peered":"2024-09-17T09:08:09.598072+0000","last_clean":"2024-09-17T09:08:09.598072+0000","last_became_active":"2024-09-17T09:07:56.259478+0000","last_became_peered":"2024-09-17T09:07:56.259478+0000","last_unstale":"2024-09-17T09:08:09.598072+0000","last_undegraded":"2024-09-17T09:08:09.598072+0000","last_fullsized":"2024-09-17T09:08:09.598072+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:15:08.191052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583617+0000","last_change":"2024-09-17T09:07:54.247538+0000","last_active":"2024-09-17T09:08:08.583617+0000","last_peered":"2024-09-17T09:08:08.583617+0000","last_clean":"2024-09-17T09:08:08.583617+0000","last_became_active":"2024-09-17T09:07:54.247258+0000","last_became_peered":"2024-09-17T09:07:54.247258+0000","last_unstale":"2024-09-17T09:08:08.583617+0000","last_undegraded":"2024-09-17T09:08:08.583617+0000","last_fullsized":"2024-09-17T09:08:08.583617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:26.043135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547539+0000","last_change":"2024-09-17T09:07:58.275169+0000","last_active":"2024-09-17T09:08:09.547539+0000","last_peered":"2024-09-17T09:08:09.547539+0000","last_clean":"2024-09-17T09:08:09.547539+0000","last_became_active":"2024-09-17T09:07:58.274877+0000","last_became_peered":"2024-09-17T09:07:58.274877+0000","last_unstale":"2024-09-17T09:08:09.547539+0000","last_undegraded":"2024-09-17T09:08:09.547539+0000","last_fullsized":"2024-09-17T09:08:09.547539+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:44.596599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596767+0000","last_change":"2024-09-17T09:08:00.277968+0000","last_active":"2024-09-17T09:08:09.596767+0000","last_peered":"2024-09-17T09:08:09.596767+0000","last_clean":"2024-09-17T09:08:09.596767+0000","last_became_active":"2024-09-17T09:08:00.277594+0000","last_became_peered":"2024-09-17T09:08:00.277594+0000","last_unstale":"2024-09-17T09:08:09.596767+0000","last_undegraded":"2024-09-17T09:08:09.596767+0000","last_fullsized":"2024-09-17T09:08:09.596767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:28:27.846574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583525+0000","last_change":"2024-09-17T09:08:00.276935+0000","last_active":"2024-09-17T09:08:08.583525+0000","last_peered":"2024-09-17T09:08:08.583525+0000","last_clean":"2024-09-17T09:08:08.583525+0000","last_became_active":"2024-09-17T09:08:00.276429+0000","last_became_peered":"2024-09-17T09:08:00.276429+0000","last_unstale":"2024-09-17T09:08:08.583525+0000","last_undegraded":"2024-09-17T09:08:08.583525+0000","last_fullsized":"2024-09-17T09:08:08.583525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:37.307114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598265+0000","last_change":"2024-09-17T09:07:54.255587+0000","last_active":"2024-09-17T09:08:09.598265+0000","last_peered":"2024-09-17T09:08:09.598265+0000","last_clean":"2024-09-17T09:08:09.598265+0000","last_became_active":"2024-09-17T09:07:54.255131+0000","last_became_peered":"2024-09-17T09:07:54.255131+0000","last_unstale":"2024-09-17T09:08:09.598265+0000","last_undegraded":"2024-09-17T09:08:09.598265+0000","last_fullsized":"2024-09-17T09:08:09.598265+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:28:56.681196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.548044+0000","last_change":"2024-09-17T09:07:56.256748+0000","last_active":"2024-09-17T09:08:09.548044+0000","last_peered":"2024-09-17T09:08:09.548044+0000","last_clean":"2024-09-17T09:08:09.548044+0000","last_became_active":"2024-09-17T09:07:56.256443+0000","last_became_peered":"2024-09-17T09:07:56.256443+0000","last_unstale":"2024-09-17T09:08:09.548044+0000","last_undegraded":"2024-09-17T09:08:09.548044+0000","last_fullsized":"2024-09-17T09:08:09.548044+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:31:43.401986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583173+0000","last_change":"2024-09-17T09:07:58.276149+0000","last_active":"2024-09-17T09:08:08.583173+0000","last_peered":"2024-09-17T09:08:08.583173+0000","last_clean":"2024-09-17T09:08:08.583173+0000","last_became_active":"2024-09-17T09:07:58.275661+0000","last_became_peered":"2024-09-17T09:07:58.275661+0000","last_unstale":"2024-09-17T09:08:08.583173+0000","last_undegraded":"2024-09-17T09:08:08.583173+0000","last_fullsized":"2024-09-17T09:08:08.583173+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:47.415050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32356,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712796,"statfs":{"total":95995035648,"available":95961903104,"internally_reserved":0,"allocated":1273856,"data_stored":855000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.59499999999999997}]},{"osd":1,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.56599999999999995}]},{"osd":2,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.505},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.46500000000000002}]},{"osd":3,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.42399999999999999}]},{"osd":4,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.58199999999999996}]},{"osd":5,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.373},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.47799999999999998}]},{"osd":6,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.44800000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28236,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716916,"statfs":{"total":95995035648,"available":95966121984,"internally_reserved":0,"allocated":1249280,"data_stored":853529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.65500000000000003}]},{"osd":1,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.625},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.64000000000000001}]},{"osd":3,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.56699999999999995}]},{"osd":4,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.33900000000000002}]},{"osd":5,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.36799999999999999}]},{"osd":7,"last update":"Tue Sep 17 09:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.41399999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31768,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713384,"statfs":{"total":95995035648,"available":95962505216,"internally_reserved":0,"allocated":667648,"data_stored":263830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64700000000000002,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.52200000000000002}]},{"osd":1,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71599999999999997,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72599999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.45500000000000002}]},{"osd":2,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.75700000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70299999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.58999999999999997}]},{"osd":3,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.749,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51600000000000001,"15min":0.51100000000000001},"min":{"1min":0.435,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74199999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.57199999999999995}]},{"osd":4,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.32200000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59999999999999998,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.38700000000000001,"15min":0.38200000000000001},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.67100000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.497}]},{"osd":6,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.48699999999999999}]},{"osd":7,"last update":"Tue Sep 17 09:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.54000000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":692224,"data_stored":265312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52000000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61199999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.495},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53700000000000003,"15min":0.54700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64100000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.63100000000000001}]},{"osd":1,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69599999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.499,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67700000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.60399999999999998}]},{"osd":2,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64800000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.65600000000000003,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.64500000000000002}]},{"osd":3,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68200000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.55000000000000004,"15min":0.55700000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67400000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61699999999999999}]},{"osd":5,"last update":"Tue Sep 17 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.34599999999999997,"15min":0.34000000000000002},"min":{"1min":0.30599999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.35599999999999998,"15min":0.34899999999999998},"min":{"1min":0.30599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.40699999999999997}]},{"osd":6,"last update":"Tue Sep 17 09:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.55600000000000005}]},{"osd":7,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.433}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31800,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713352,"statfs":{"total":95995035648,"available":95962472448,"internally_reserved":0,"allocated":700416,"data_stored":265224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.379,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.48499999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.35499999999999998}]},{"osd":1,"last update":"Tue Sep 17 09:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41299999999999998,"15min":0.41899999999999998},"min":{"1min":0.29399999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.39800000000000002,"15min":0.41499999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.48599999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.33600000000000002}]},{"osd":2,"last update":"Tue Sep 17 09:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41699999999999998,"15min":0.42499999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.52700000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.40500000000000003,"15min":0.41099999999999998},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.57099999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.45800000000000002}]},{"osd":4,"last update":"Tue Sep 17 09:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.52200000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.52500000000000002}]},{"osd":5,"last update":"Tue Sep 17 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38200000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.436},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52500000000000002,"15min":0.52000000000000002},"min":{"1min":0.41799999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.57699999999999996}]},{"osd":6,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51400000000000001}]},{"osd":7,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.48899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31748,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713404,"statfs":{"total":95995035648,"available":95962525696,"internally_reserved":0,"allocated":647168,"data_stored":263060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36499999999999999,"15min":0.34999999999999998},"min":{"1min":0.253,"5min":0.25,"15min":0.25},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.375,"15min":0.36299999999999999},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.40699999999999997}]},{"osd":1,"last update":"Tue Sep 17 09:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36299999999999999,"15min":0.35599999999999998},"min":{"1min":0.23599999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.41599999999999998,"15min":0.40300000000000002},"min":{"1min":0.28699999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.42099999999999999}]},{"osd":3,"last update":"Tue Sep 17 09:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.315,"15min":0.315},"max":{"1min":0.59999999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.371},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.433,"15min":0.435},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.45100000000000001}]},{"osd":4,"last update":"Tue Sep 17 09:08:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61699999999999999,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.51200000000000001}]},{"osd":5,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.46999999999999997,"5min":0.441,"15min":0.441},"max":{"1min":0.89600000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.60399999999999998,"15min":0.61099999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.503}]},{"osd":6,"last update":"Tue Sep 17 09:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.53100000000000003}]},{"osd":7,"last update":"Tue Sep 17 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.61099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31812,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713340,"statfs":{"total":95995035648,"available":95962460160,"internally_reserved":0,"allocated":712704,"data_stored":265513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32400000000000001,"15min":0.30099999999999999},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.44900000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33200000000000002,"15min":0.313},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.495,"5min":0.496,"15min":0.496},"last":0.35399999999999998}]},{"osd":2,"last update":"Tue Sep 17 09:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36799999999999999,"15min":0.36099999999999999},"min":{"1min":0.23400000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.35499999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.28899999999999998}]},{"osd":3,"last update":"Tue Sep 17 09:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.45200000000000001,"15min":0.46000000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.58299999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.37},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.45400000000000001,"15min":0.46200000000000002},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.58199999999999996,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.34200000000000003}]},{"osd":4,"last update":"Tue Sep 17 09:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53500000000000003}]},{"osd":5,"last update":"Tue Sep 17 09:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.51400000000000001}]},{"osd":6,"last update":"Tue Sep 17 09:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.496},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59699999999999998}]},{"osd":7,"last update":"Tue Sep 17 09:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.621}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32360,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712792,"statfs":{"total":95995035648,"available":95961899008,"internally_reserved":0,"allocated":1277952,"data_stored":855028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33600000000000002,"15min":0.32300000000000001},"min":{"1min":0.23799999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.53700000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.308},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.35099999999999998,"15min":0.33500000000000002},"min":{"1min":0.24199999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.51000000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.26000000000000001}]},{"osd":2,"last update":"Tue Sep 17 09:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51700000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.373,"15min":0.371},"min":{"1min":0.28899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.51000000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.34899999999999998}]},{"osd":3,"last update":"Tue Sep 17 09:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.41299999999999998,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52800000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.40400000000000003,"15min":0.41599999999999998},"min":{"1min":0.26300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.53600000000000003,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38500000000000001}]},{"osd":4,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54500000000000004,"15min":0.54200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.59399999999999997}]},{"osd":5,"last update":"Tue Sep 17 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60499999999999998}]},{"osd":6,"last update":"Tue Sep 17 09:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.57099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T09:09:15.589 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T09:09:15.784 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:09:16.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:16 smithi173 ceph-mon[33421]: from='client.24584 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:09:16.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:16 smithi173 ceph-mon[33421]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:16.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:16 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:16.757 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:16 smithi079 ceph-mon[29577]: from='client.24584 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:09:16.758 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:16 smithi079 ceph-mon[29577]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:16.758 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:16 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:16.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:16 smithi079 ceph-mon[40839]: from='client.24584 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:09:16.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:16 smithi079 ceph-mon[40839]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:16.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:16 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:17.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:17 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:17.770 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:09:17.771 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:17 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:17.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:17 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:17.776 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-09-17T09:09:18.625 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-17T09:09:17.367287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":998,"num_read_kb":844,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":597,"ondisk_log_size":597,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":247776,"kb_used_data":7052,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":749713440,"statfs":{"total":767960285184,"available":767706562560,"internally_reserved":0,"allocated":7221248,"data_stored":3886496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":246402617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002685"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583546+0000","last_change":"2024-09-17T09:07:54.251646+0000","last_active":"2024-09-17T09:08:08.583546+0000","last_peered":"2024-09-17T09:08:08.583546+0000","last_clean":"2024-09-17T09:08:08.583546+0000","last_became_active":"2024-09-17T09:07:54.251361+0000","last_became_peered":"2024-09-17T09:07:54.251361+0000","last_unstale":"2024-09-17T09:08:08.583546+0000","last_undegraded":"2024-09-17T09:08:08.583546+0000","last_fullsized":"2024-09-17T09:08:08.583546+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:31.298221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596998+0000","last_change":"2024-09-17T09:07:56.261459+0000","last_active":"2024-09-17T09:08:09.596998+0000","last_peered":"2024-09-17T09:08:09.596998+0000","last_clean":"2024-09-17T09:08:09.596998+0000","last_became_active":"2024-09-17T09:07:56.261158+0000","last_became_peered":"2024-09-17T09:07:56.261158+0000","last_unstale":"2024-09-17T09:08:09.596998+0000","last_undegraded":"2024-09-17T09:08:09.596998+0000","last_fullsized":"2024-09-17T09:08:09.596998+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:22.307427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583644+0000","last_change":"2024-09-17T09:07:58.272427+0000","last_active":"2024-09-17T09:08:08.583644+0000","last_peered":"2024-09-17T09:08:08.583644+0000","last_clean":"2024-09-17T09:08:08.583644+0000","last_became_active":"2024-09-17T09:07:58.272167+0000","last_became_peered":"2024-09-17T09:07:58.272167+0000","last_unstale":"2024-09-17T09:08:08.583644+0000","last_undegraded":"2024-09-17T09:08:08.583644+0000","last_fullsized":"2024-09-17T09:08:08.583644+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:54.033779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597091+0000","last_change":"2024-09-17T09:08:00.278458+0000","last_active":"2024-09-17T09:08:09.597091+0000","last_peered":"2024-09-17T09:08:09.597091+0000","last_clean":"2024-09-17T09:08:09.597091+0000","last_became_active":"2024-09-17T09:08:00.278188+0000","last_became_peered":"2024-09-17T09:08:00.278188+0000","last_unstale":"2024-09-17T09:08:09.597091+0000","last_undegraded":"2024-09-17T09:08:09.597091+0000","last_fullsized":"2024-09-17T09:08:09.597091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:42.543582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597677+0000","last_change":"2024-09-17T09:08:00.413079+0000","last_active":"2024-09-17T09:08:09.597677+0000","last_peered":"2024-09-17T09:08:09.597677+0000","last_clean":"2024-09-17T09:08:09.597677+0000","last_became_active":"2024-09-17T09:08:00.412524+0000","last_became_peered":"2024-09-17T09:08:00.412524+0000","last_unstale":"2024-09-17T09:08:09.597677+0000","last_undegraded":"2024-09-17T09:08:09.597677+0000","last_fullsized":"2024-09-17T09:08:09.597677+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:07.023236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598555+0000","last_change":"2024-09-17T09:07:54.250825+0000","last_active":"2024-09-17T09:08:09.598555+0000","last_peered":"2024-09-17T09:08:09.598555+0000","last_clean":"2024-09-17T09:08:09.598555+0000","last_became_active":"2024-09-17T09:07:54.250607+0000","last_became_peered":"2024-09-17T09:07:54.250607+0000","last_unstale":"2024-09-17T09:08:09.598555+0000","last_undegraded":"2024-09-17T09:08:09.598555+0000","last_fullsized":"2024-09-17T09:08:09.598555+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:42:09.304832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597636+0000","last_change":"2024-09-17T09:07:56.248032+0000","last_active":"2024-09-17T09:08:09.597636+0000","last_peered":"2024-09-17T09:08:09.597636+0000","last_clean":"2024-09-17T09:08:09.597636+0000","last_became_active":"2024-09-17T09:07:56.247566+0000","last_became_peered":"2024-09-17T09:07:56.247566+0000","last_unstale":"2024-09-17T09:08:09.597636+0000","last_undegraded":"2024-09-17T09:08:09.597636+0000","last_fullsized":"2024-09-17T09:08:09.597636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:10:05.758271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596767+0000","last_change":"2024-09-17T09:07:58.269563+0000","last_active":"2024-09-17T09:08:09.596767+0000","last_peered":"2024-09-17T09:08:09.596767+0000","last_clean":"2024-09-17T09:08:09.596767+0000","last_became_active":"2024-09-17T09:07:58.269291+0000","last_became_peered":"2024-09-17T09:07:58.269291+0000","last_unstale":"2024-09-17T09:08:09.596767+0000","last_undegraded":"2024-09-17T09:08:09.596767+0000","last_fullsized":"2024-09-17T09:08:09.596767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:51:14.556764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582720+0000","last_change":"2024-09-17T09:07:54.249208+0000","last_active":"2024-09-17T09:08:08.582720+0000","last_peered":"2024-09-17T09:08:08.582720+0000","last_clean":"2024-09-17T09:08:08.582720+0000","last_became_active":"2024-09-17T09:07:54.248955+0000","last_became_peered":"2024-09-17T09:07:54.248955+0000","last_unstale":"2024-09-17T09:08:08.582720+0000","last_undegraded":"2024-09-17T09:08:08.582720+0000","last_fullsized":"2024-09-17T09:08:08.582720+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:49:57.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597439+0000","last_change":"2024-09-17T09:07:56.252608+0000","last_active":"2024-09-17T09:08:09.597439+0000","last_peered":"2024-09-17T09:08:09.597439+0000","last_clean":"2024-09-17T09:08:09.597439+0000","last_became_active":"2024-09-17T09:07:56.252371+0000","last_became_peered":"2024-09-17T09:07:56.252371+0000","last_unstale":"2024-09-17T09:08:09.597439+0000","last_undegraded":"2024-09-17T09:08:09.597439+0000","last_fullsized":"2024-09-17T09:08:09.597439+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:49.302726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582314+0000","last_change":"2024-09-17T09:07:58.277397+0000","last_active":"2024-09-17T09:08:08.582314+0000","last_peered":"2024-09-17T09:08:08.582314+0000","last_clean":"2024-09-17T09:08:08.582314+0000","last_became_active":"2024-09-17T09:07:58.277138+0000","last_became_peered":"2024-09-17T09:07:58.277138+0000","last_unstale":"2024-09-17T09:08:08.582314+0000","last_undegraded":"2024-09-17T09:08:08.582314+0000","last_fullsized":"2024-09-17T09:08:08.582314+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:40.348729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582553+0000","last_change":"2024-09-17T09:08:00.280989+0000","last_active":"2024-09-17T09:08:08.582553+0000","last_peered":"2024-09-17T09:08:08.582553+0000","last_clean":"2024-09-17T09:08:08.582553+0000","last_became_active":"2024-09-17T09:08:00.280677+0000","last_became_peered":"2024-09-17T09:08:00.280677+0000","last_unstale":"2024-09-17T09:08:08.582553+0000","last_undegraded":"2024-09-17T09:08:08.582553+0000","last_fullsized":"2024-09-17T09:08:08.582553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:09:46.012224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582398+0000","last_change":"2024-09-17T09:07:54.247832+0000","last_active":"2024-09-17T09:08:08.582398+0000","last_peered":"2024-09-17T09:08:08.582398+0000","last_clean":"2024-09-17T09:08:08.582398+0000","last_became_active":"2024-09-17T09:07:54.247560+0000","last_became_peered":"2024-09-17T09:07:54.247560+0000","last_unstale":"2024-09-17T09:08:08.582398+0000","last_undegraded":"2024-09-17T09:08:08.582398+0000","last_fullsized":"2024-09-17T09:08:08.582398+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:17:39.763847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":27,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596936+0000","last_change":"2024-09-17T09:07:56.261559+0000","last_active":"2024-09-17T09:08:09.596936+0000","last_peered":"2024-09-17T09:08:09.596936+0000","last_clean":"2024-09-17T09:08:09.596936+0000","last_became_active":"2024-09-17T09:07:56.261268+0000","last_became_peered":"2024-09-17T09:07:56.261268+0000","last_unstale":"2024-09-17T09:08:09.596936+0000","last_undegraded":"2024-09-17T09:08:09.596936+0000","last_fullsized":"2024-09-17T09:08:09.596936+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:26.542959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596854+0000","last_change":"2024-09-17T09:07:58.280390+0000","last_active":"2024-09-17T09:08:09.596854+0000","last_peered":"2024-09-17T09:08:09.596854+0000","last_clean":"2024-09-17T09:08:09.596854+0000","last_became_active":"2024-09-17T09:07:58.279948+0000","last_became_peered":"2024-09-17T09:07:58.279948+0000","last_unstale":"2024-09-17T09:08:09.596854+0000","last_undegraded":"2024-09-17T09:08:09.596854+0000","last_fullsized":"2024-09-17T09:08:09.596854+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:46.715628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582307+0000","last_change":"2024-09-17T09:08:00.282477+0000","last_active":"2024-09-17T09:08:08.582307+0000","last_peered":"2024-09-17T09:08:08.582307+0000","last_clean":"2024-09-17T09:08:08.582307+0000","last_became_active":"2024-09-17T09:08:00.282170+0000","last_became_peered":"2024-09-17T09:08:00.282170+0000","last_unstale":"2024-09-17T09:08:08.582307+0000","last_undegraded":"2024-09-17T09:08:08.582307+0000","last_fullsized":"2024-09-17T09:08:08.582307+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:57.183905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598016+0000","last_change":"2024-09-17T09:08:00.410688+0000","last_active":"2024-09-17T09:08:09.598016+0000","last_peered":"2024-09-17T09:08:09.598016+0000","last_clean":"2024-09-17T09:08:09.598016+0000","last_became_active":"2024-09-17T09:08:00.410344+0000","last_became_peered":"2024-09-17T09:08:00.410344+0000","last_unstale":"2024-09-17T09:08:09.598016+0000","last_undegraded":"2024-09-17T09:08:09.598016+0000","last_fullsized":"2024-09-17T09:08:09.598016+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:49:48.873644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"60'1","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584690+0000","last_change":"2024-09-17T09:07:54.248415+0000","last_active":"2024-09-17T09:08:08.584690+0000","last_peered":"2024-09-17T09:08:08.584690+0000","last_clean":"2024-09-17T09:08:08.584690+0000","last_became_active":"2024-09-17T09:07:54.248140+0000","last_became_peered":"2024-09-17T09:07:54.248140+0000","last_unstale":"2024-09-17T09:08:08.584690+0000","last_undegraded":"2024-09-17T09:08:08.584690+0000","last_fullsized":"2024-09-17T09:08:08.584690+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:35.469508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597660+0000","last_change":"2024-09-17T09:07:56.257098+0000","last_active":"2024-09-17T09:08:09.597660+0000","last_peered":"2024-09-17T09:08:09.597660+0000","last_clean":"2024-09-17T09:08:09.597660+0000","last_became_active":"2024-09-17T09:07:56.256823+0000","last_became_peered":"2024-09-17T09:07:56.256823+0000","last_unstale":"2024-09-17T09:08:09.597660+0000","last_undegraded":"2024-09-17T09:08:09.597660+0000","last_fullsized":"2024-09-17T09:08:09.597660+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:26:19.900924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583797+0000","last_change":"2024-09-17T09:07:58.278639+0000","last_active":"2024-09-17T09:08:08.583797+0000","last_peered":"2024-09-17T09:08:08.583797+0000","last_clean":"2024-09-17T09:08:08.583797+0000","last_became_active":"2024-09-17T09:07:58.278310+0000","last_became_peered":"2024-09-17T09:07:58.278310+0000","last_unstale":"2024-09-17T09:08:08.583797+0000","last_undegraded":"2024-09-17T09:08:08.583797+0000","last_fullsized":"2024-09-17T09:08:08.583797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:20.244724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598630+0000","last_change":"2024-09-17T09:08:00.411157+0000","last_active":"2024-09-17T09:08:09.598630+0000","last_peered":"2024-09-17T09:08:09.598630+0000","last_clean":"2024-09-17T09:08:09.598630+0000","last_became_active":"2024-09-17T09:08:00.410740+0000","last_became_peered":"2024-09-17T09:08:00.410740+0000","last_unstale":"2024-09-17T09:08:09.598630+0000","last_undegraded":"2024-09-17T09:08:09.598630+0000","last_fullsized":"2024-09-17T09:08:09.598630+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:44:55.092423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597572+0000","last_change":"2024-09-17T09:07:54.239863+0000","last_active":"2024-09-17T09:08:09.597572+0000","last_peered":"2024-09-17T09:08:09.597572+0000","last_clean":"2024-09-17T09:08:09.597572+0000","last_became_active":"2024-09-17T09:07:54.239545+0000","last_became_peered":"2024-09-17T09:07:54.239545+0000","last_unstale":"2024-09-17T09:08:09.597572+0000","last_undegraded":"2024-09-17T09:08:09.597572+0000","last_fullsized":"2024-09-17T09:08:09.597572+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:18:12.769978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":40,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598526+0000","last_change":"2024-09-17T09:07:56.257316+0000","last_active":"2024-09-17T09:08:09.598526+0000","last_peered":"2024-09-17T09:08:09.598526+0000","last_clean":"2024-09-17T09:08:09.598526+0000","last_became_active":"2024-09-17T09:07:56.257088+0000","last_became_peered":"2024-09-17T09:07:56.257088+0000","last_unstale":"2024-09-17T09:08:09.598526+0000","last_undegraded":"2024-09-17T09:08:09.598526+0000","last_fullsized":"2024-09-17T09:08:09.598526+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:37:26.810507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596971+0000","last_change":"2024-09-17T09:07:58.274279+0000","last_active":"2024-09-17T09:08:09.596971+0000","last_peered":"2024-09-17T09:08:09.596971+0000","last_clean":"2024-09-17T09:08:09.596971+0000","last_became_active":"2024-09-17T09:07:58.273485+0000","last_became_peered":"2024-09-17T09:07:58.273485+0000","last_unstale":"2024-09-17T09:08:09.596971+0000","last_undegraded":"2024-09-17T09:08:09.596971+0000","last_fullsized":"2024-09-17T09:08:09.596971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:39.456511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597336+0000","last_change":"2024-09-17T09:08:00.277959+0000","last_active":"2024-09-17T09:08:09.597336+0000","last_peered":"2024-09-17T09:08:09.597336+0000","last_clean":"2024-09-17T09:08:09.597336+0000","last_became_active":"2024-09-17T09:08:00.277650+0000","last_became_peered":"2024-09-17T09:08:00.277650+0000","last_unstale":"2024-09-17T09:08:09.597336+0000","last_undegraded":"2024-09-17T09:08:09.597336+0000","last_fullsized":"2024-09-17T09:08:09.597336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:57.057584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":24,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583808+0000","last_change":"2024-09-17T09:07:54.254871+0000","last_active":"2024-09-17T09:08:08.583808+0000","last_peered":"2024-09-17T09:08:08.583808+0000","last_clean":"2024-09-17T09:08:08.583808+0000","last_became_active":"2024-09-17T09:07:54.254555+0000","last_became_peered":"2024-09-17T09:07:54.254555+0000","last_unstale":"2024-09-17T09:08:08.583808+0000","last_undegraded":"2024-09-17T09:08:08.583808+0000","last_fullsized":"2024-09-17T09:08:08.583808+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:14:14.543065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583337+0000","last_change":"2024-09-17T09:07:56.254113+0000","last_active":"2024-09-17T09:08:08.583337+0000","last_peered":"2024-09-17T09:08:08.583337+0000","last_clean":"2024-09-17T09:08:08.583337+0000","last_became_active":"2024-09-17T09:07:56.253610+0000","last_became_peered":"2024-09-17T09:07:56.253610+0000","last_unstale":"2024-09-17T09:08:08.583337+0000","last_undegraded":"2024-09-17T09:08:08.583337+0000","last_fullsized":"2024-09-17T09:08:08.583337+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:51:00.410902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":42,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:07.704939+0000","last_change":"2024-09-17T09:07:58.274291+0000","last_active":"2024-09-17T09:09:07.704939+0000","last_peered":"2024-09-17T09:09:07.704939+0000","last_clean":"2024-09-17T09:09:07.704939+0000","last_became_active":"2024-09-17T09:07:58.273941+0000","last_became_peered":"2024-09-17T09:07:58.273941+0000","last_unstale":"2024-09-17T09:09:07.704939+0000","last_undegraded":"2024-09-17T09:09:07.704939+0000","last_fullsized":"2024-09-17T09:09:07.704939+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:48:56.584346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584451+0000","last_change":"2024-09-17T09:07:56.262979+0000","last_active":"2024-09-17T09:08:08.584451+0000","last_peered":"2024-09-17T09:08:08.584451+0000","last_clean":"2024-09-17T09:08:08.584451+0000","last_became_active":"2024-09-17T09:07:56.262580+0000","last_became_peered":"2024-09-17T09:07:56.262580+0000","last_unstale":"2024-09-17T09:08:08.584451+0000","last_undegraded":"2024-09-17T09:08:08.584451+0000","last_fullsized":"2024-09-17T09:08:08.584451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:20:31.479926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598345+0000","last_change":"2024-09-17T09:07:54.249565+0000","last_active":"2024-09-17T09:08:09.598345+0000","last_peered":"2024-09-17T09:08:09.598345+0000","last_clean":"2024-09-17T09:08:09.598345+0000","last_became_active":"2024-09-17T09:07:54.249322+0000","last_became_peered":"2024-09-17T09:07:54.249322+0000","last_unstale":"2024-09-17T09:08:09.598345+0000","last_undegraded":"2024-09-17T09:08:09.598345+0000","last_fullsized":"2024-09-17T09:08:09.598345+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:43.618527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":39,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:07.704943+0000","last_change":"2024-09-17T09:07:58.274124+0000","last_active":"2024-09-17T09:09:07.704943+0000","last_peered":"2024-09-17T09:09:07.704943+0000","last_clean":"2024-09-17T09:09:07.704943+0000","last_became_active":"2024-09-17T09:07:58.273476+0000","last_became_peered":"2024-09-17T09:07:58.273476+0000","last_unstale":"2024-09-17T09:09:07.704943+0000","last_undegraded":"2024-09-17T09:09:07.704943+0000","last_fullsized":"2024-09-17T09:09:07.704943+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:16.196820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582550+0000","last_change":"2024-09-17T09:08:00.277346+0000","last_active":"2024-09-17T09:08:08.582550+0000","last_peered":"2024-09-17T09:08:08.582550+0000","last_clean":"2024-09-17T09:08:08.582550+0000","last_became_active":"2024-09-17T09:08:00.277068+0000","last_became_peered":"2024-09-17T09:08:00.277068+0000","last_unstale":"2024-09-17T09:08:08.582550+0000","last_undegraded":"2024-09-17T09:08:08.582550+0000","last_fullsized":"2024-09-17T09:08:08.582550+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:03.417308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":49,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598072+0000","last_change":"2024-09-17T09:07:56.248310+0000","last_active":"2024-09-17T09:08:09.598072+0000","last_peered":"2024-09-17T09:08:09.598072+0000","last_clean":"2024-09-17T09:08:09.598072+0000","last_became_active":"2024-09-17T09:07:56.247999+0000","last_became_peered":"2024-09-17T09:07:56.247999+0000","last_unstale":"2024-09-17T09:08:09.598072+0000","last_undegraded":"2024-09-17T09:08:09.598072+0000","last_fullsized":"2024-09-17T09:08:09.598072+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:21.833823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598831+0000","last_change":"2024-09-17T09:07:54.248020+0000","last_active":"2024-09-17T09:08:09.598831+0000","last_peered":"2024-09-17T09:08:09.598831+0000","last_clean":"2024-09-17T09:08:09.598831+0000","last_became_active":"2024-09-17T09:07:54.247792+0000","last_became_peered":"2024-09-17T09:07:54.247792+0000","last_unstale":"2024-09-17T09:08:09.598831+0000","last_undegraded":"2024-09-17T09:08:09.598831+0000","last_fullsized":"2024-09-17T09:08:09.598831+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:08:58.091716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597615+0000","last_change":"2024-09-17T09:07:58.273144+0000","last_active":"2024-09-17T09:08:09.597615+0000","last_peered":"2024-09-17T09:08:09.597615+0000","last_clean":"2024-09-17T09:08:09.597615+0000","last_became_active":"2024-09-17T09:07:58.272850+0000","last_became_peered":"2024-09-17T09:07:58.272850+0000","last_unstale":"2024-09-17T09:08:09.597615+0000","last_undegraded":"2024-09-17T09:08:09.597615+0000","last_fullsized":"2024-09-17T09:08:09.597615+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:09.700309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597486+0000","last_change":"2024-09-17T09:08:00.269002+0000","last_active":"2024-09-17T09:08:09.597486+0000","last_peered":"2024-09-17T09:08:09.597486+0000","last_clean":"2024-09-17T09:08:09.597486+0000","last_became_active":"2024-09-17T09:08:00.268708+0000","last_became_peered":"2024-09-17T09:08:00.268708+0000","last_unstale":"2024-09-17T09:08:09.597486+0000","last_undegraded":"2024-09-17T09:08:09.597486+0000","last_fullsized":"2024-09-17T09:08:09.597486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:57.482275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":38,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596847+0000","last_change":"2024-09-17T09:07:56.250184+0000","last_active":"2024-09-17T09:08:09.596847+0000","last_peered":"2024-09-17T09:08:09.596847+0000","last_clean":"2024-09-17T09:08:09.596847+0000","last_became_active":"2024-09-17T09:07:56.249903+0000","last_became_peered":"2024-09-17T09:07:56.249903+0000","last_unstale":"2024-09-17T09:08:09.596847+0000","last_undegraded":"2024-09-17T09:08:09.596847+0000","last_fullsized":"2024-09-17T09:08:09.596847+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:59.727703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583615+0000","last_change":"2024-09-17T09:07:54.245488+0000","last_active":"2024-09-17T09:08:08.583615+0000","last_peered":"2024-09-17T09:08:08.583615+0000","last_clean":"2024-09-17T09:08:08.583615+0000","last_became_active":"2024-09-17T09:07:54.245095+0000","last_became_peered":"2024-09-17T09:07:54.245095+0000","last_unstale":"2024-09-17T09:08:08.583615+0000","last_undegraded":"2024-09-17T09:08:08.583615+0000","last_fullsized":"2024-09-17T09:08:08.583615+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:27.984879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":39,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:07.704702+0000","last_change":"2024-09-17T09:07:58.273778+0000","last_active":"2024-09-17T09:09:07.704702+0000","last_peered":"2024-09-17T09:09:07.704702+0000","last_clean":"2024-09-17T09:09:07.704702+0000","last_became_active":"2024-09-17T09:07:58.273431+0000","last_became_peered":"2024-09-17T09:07:58.273431+0000","last_unstale":"2024-09-17T09:09:07.704702+0000","last_undegraded":"2024-09-17T09:09:07.704702+0000","last_fullsized":"2024-09-17T09:09:07.704702+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:41:48.911678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596846+0000","last_change":"2024-09-17T09:08:00.411687+0000","last_active":"2024-09-17T09:08:09.596846+0000","last_peered":"2024-09-17T09:08:09.596846+0000","last_clean":"2024-09-17T09:08:09.596846+0000","last_became_active":"2024-09-17T09:08:00.411346+0000","last_became_peered":"2024-09-17T09:08:00.411346+0000","last_unstale":"2024-09-17T09:08:09.596846+0000","last_undegraded":"2024-09-17T09:08:09.596846+0000","last_fullsized":"2024-09-17T09:08:09.596846+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:48.090997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583101+0000","last_change":"2024-09-17T09:07:56.263388+0000","last_active":"2024-09-17T09:08:08.583101+0000","last_peered":"2024-09-17T09:08:08.583101+0000","last_clean":"2024-09-17T09:08:08.583101+0000","last_became_active":"2024-09-17T09:07:56.263032+0000","last_became_peered":"2024-09-17T09:07:56.263032+0000","last_unstale":"2024-09-17T09:08:08.583101+0000","last_undegraded":"2024-09-17T09:08:08.583101+0000","last_fullsized":"2024-09-17T09:08:08.583101+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:50:46.023755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582833+0000","last_change":"2024-09-17T09:07:54.250681+0000","last_active":"2024-09-17T09:08:08.582833+0000","last_peered":"2024-09-17T09:08:08.582833+0000","last_clean":"2024-09-17T09:08:08.582833+0000","last_became_active":"2024-09-17T09:07:54.250393+0000","last_became_peered":"2024-09-17T09:07:54.250393+0000","last_unstale":"2024-09-17T09:08:08.582833+0000","last_undegraded":"2024-09-17T09:08:08.582833+0000","last_fullsized":"2024-09-17T09:08:08.582833+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:27:11.495491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":42,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:07.704721+0000","last_change":"2024-09-17T09:07:58.280766+0000","last_active":"2024-09-17T09:09:07.704721+0000","last_peered":"2024-09-17T09:09:07.704721+0000","last_clean":"2024-09-17T09:09:07.704721+0000","last_became_active":"2024-09-17T09:07:58.280431+0000","last_became_peered":"2024-09-17T09:07:58.280431+0000","last_unstale":"2024-09-17T09:09:07.704721+0000","last_undegraded":"2024-09-17T09:09:07.704721+0000","last_fullsized":"2024-09-17T09:09:07.704721+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:43.136981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582666+0000","last_change":"2024-09-17T09:08:00.282626+0000","last_active":"2024-09-17T09:08:08.582666+0000","last_peered":"2024-09-17T09:08:08.582666+0000","last_clean":"2024-09-17T09:08:08.582666+0000","last_became_active":"2024-09-17T09:08:00.282326+0000","last_became_peered":"2024-09-17T09:08:00.282326+0000","last_unstale":"2024-09-17T09:08:08.582666+0000","last_undegraded":"2024-09-17T09:08:08.582666+0000","last_fullsized":"2024-09-17T09:08:08.582666+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:31:43.763578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":53,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584576+0000","last_change":"2024-09-17T09:07:56.252301+0000","last_active":"2024-09-17T09:08:08.584576+0000","last_peered":"2024-09-17T09:08:08.584576+0000","last_clean":"2024-09-17T09:08:08.584576+0000","last_became_active":"2024-09-17T09:07:56.252027+0000","last_became_peered":"2024-09-17T09:07:56.252027+0000","last_unstale":"2024-09-17T09:08:08.584576+0000","last_undegraded":"2024-09-17T09:08:08.584576+0000","last_fullsized":"2024-09-17T09:08:08.584576+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:10.437759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584305+0000","last_change":"2024-09-17T09:07:54.243714+0000","last_active":"2024-09-17T09:08:08.584305+0000","last_peered":"2024-09-17T09:08:08.584305+0000","last_clean":"2024-09-17T09:08:08.584305+0000","last_became_active":"2024-09-17T09:07:54.243433+0000","last_became_peered":"2024-09-17T09:07:54.243433+0000","last_unstale":"2024-09-17T09:08:08.584305+0000","last_undegraded":"2024-09-17T09:08:08.584305+0000","last_fullsized":"2024-09-17T09:08:08.584305+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:53.527157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547588+0000","last_change":"2024-09-17T09:07:58.276721+0000","last_active":"2024-09-17T09:08:09.547588+0000","last_peered":"2024-09-17T09:08:09.547588+0000","last_clean":"2024-09-17T09:08:09.547588+0000","last_became_active":"2024-09-17T09:07:58.276440+0000","last_became_peered":"2024-09-17T09:07:58.276440+0000","last_unstale":"2024-09-17T09:08:09.547588+0000","last_undegraded":"2024-09-17T09:08:09.547588+0000","last_fullsized":"2024-09-17T09:08:09.547588+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:47:12.201120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583566+0000","last_change":"2024-09-17T09:08:00.410764+0000","last_active":"2024-09-17T09:08:08.583566+0000","last_peered":"2024-09-17T09:08:08.583566+0000","last_clean":"2024-09-17T09:08:08.583566+0000","last_became_active":"2024-09-17T09:08:00.410377+0000","last_became_peered":"2024-09-17T09:08:00.410377+0000","last_unstale":"2024-09-17T09:08:08.583566+0000","last_undegraded":"2024-09-17T09:08:08.583566+0000","last_fullsized":"2024-09-17T09:08:08.583566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:50:20.460707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":81,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:07.704739+0000","last_change":"2024-09-17T09:07:56.261870+0000","last_active":"2024-09-17T09:09:07.704739+0000","last_peered":"2024-09-17T09:09:07.704739+0000","last_clean":"2024-09-17T09:09:07.704739+0000","last_became_active":"2024-09-17T09:07:56.261549+0000","last_became_peered":"2024-09-17T09:07:56.261549+0000","last_unstale":"2024-09-17T09:09:07.704739+0000","last_undegraded":"2024-09-17T09:09:07.704739+0000","last_fullsized":"2024-09-17T09:09:07.704739+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:11:08.133092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597518+0000","last_change":"2024-09-17T09:07:54.249306+0000","last_active":"2024-09-17T09:08:09.597518+0000","last_peered":"2024-09-17T09:08:09.597518+0000","last_clean":"2024-09-17T09:08:09.597518+0000","last_became_active":"2024-09-17T09:07:54.248862+0000","last_became_peered":"2024-09-17T09:07:54.248862+0000","last_unstale":"2024-09-17T09:08:09.597518+0000","last_undegraded":"2024-09-17T09:08:09.597518+0000","last_fullsized":"2024-09-17T09:08:09.597518+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:04.247970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":26,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582494+0000","last_change":"2024-09-17T09:07:58.272009+0000","last_active":"2024-09-17T09:08:08.582494+0000","last_peered":"2024-09-17T09:08:08.582494+0000","last_clean":"2024-09-17T09:08:08.582494+0000","last_became_active":"2024-09-17T09:07:54.248873+0000","last_became_peered":"2024-09-17T09:07:54.248873+0000","last_unstale":"2024-09-17T09:08:08.582494+0000","last_undegraded":"2024-09-17T09:08:08.582494+0000","last_fullsized":"2024-09-17T09:08:08.582494+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:04.719857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019774290000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583221+0000","last_change":"2024-09-17T09:07:58.278978+0000","last_active":"2024-09-17T09:08:08.583221+0000","last_peered":"2024-09-17T09:08:08.583221+0000","last_clean":"2024-09-17T09:08:08.583221+0000","last_became_active":"2024-09-17T09:07:58.278652+0000","last_became_peered":"2024-09-17T09:07:58.278652+0000","last_unstale":"2024-09-17T09:08:08.583221+0000","last_undegraded":"2024-09-17T09:08:08.583221+0000","last_fullsized":"2024-09-17T09:08:08.583221+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:05:23.314898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598350+0000","last_change":"2024-09-17T09:08:00.275542+0000","last_active":"2024-09-17T09:08:09.598350+0000","last_peered":"2024-09-17T09:08:09.598350+0000","last_clean":"2024-09-17T09:08:09.598350+0000","last_became_active":"2024-09-17T09:08:00.275267+0000","last_became_peered":"2024-09-17T09:08:00.275267+0000","last_unstale":"2024-09-17T09:08:09.598350+0000","last_undegraded":"2024-09-17T09:08:09.598350+0000","last_fullsized":"2024-09-17T09:08:09.598350+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:57.874829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":44,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584462+0000","last_change":"2024-09-17T09:07:56.262497+0000","last_active":"2024-09-17T09:08:08.584462+0000","last_peered":"2024-09-17T09:08:08.584462+0000","last_clean":"2024-09-17T09:08:08.584462+0000","last_became_active":"2024-09-17T09:07:56.262177+0000","last_became_peered":"2024-09-17T09:07:56.262177+0000","last_unstale":"2024-09-17T09:08:08.584462+0000","last_undegraded":"2024-09-17T09:08:08.584462+0000","last_fullsized":"2024-09-17T09:08:08.584462+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:22:31.229991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583435+0000","last_change":"2024-09-17T09:07:54.249798+0000","last_active":"2024-09-17T09:08:08.583435+0000","last_peered":"2024-09-17T09:08:08.583435+0000","last_clean":"2024-09-17T09:08:08.583435+0000","last_became_active":"2024-09-17T09:07:54.249520+0000","last_became_peered":"2024-09-17T09:07:54.249520+0000","last_unstale":"2024-09-17T09:08:08.583435+0000","last_undegraded":"2024-09-17T09:08:08.583435+0000","last_fullsized":"2024-09-17T09:08:08.583435+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:55.684631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":25,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597641+0000","last_change":"2024-09-17T09:07:58.263960+0000","last_active":"2024-09-17T09:08:09.597641+0000","last_peered":"2024-09-17T09:08:09.597641+0000","last_clean":"2024-09-17T09:08:09.597641+0000","last_became_active":"2024-09-17T09:07:54.250843+0000","last_became_peered":"2024-09-17T09:07:54.250843+0000","last_unstale":"2024-09-17T09:08:09.597641+0000","last_undegraded":"2024-09-17T09:08:09.597641+0000","last_fullsized":"2024-09-17T09:08:09.597641+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:24:22.827498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001848487,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597598+0000","last_change":"2024-09-17T09:07:58.273296+0000","last_active":"2024-09-17T09:08:09.597598+0000","last_peered":"2024-09-17T09:08:09.597598+0000","last_clean":"2024-09-17T09:08:09.597598+0000","last_became_active":"2024-09-17T09:07:58.273005+0000","last_became_peered":"2024-09-17T09:07:58.273005+0000","last_unstale":"2024-09-17T09:08:09.597598+0000","last_undegraded":"2024-09-17T09:08:09.597598+0000","last_fullsized":"2024-09-17T09:08:09.597598+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:16:29.679121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597800+0000","last_change":"2024-09-17T09:08:00.410953+0000","last_active":"2024-09-17T09:08:09.597800+0000","last_peered":"2024-09-17T09:08:09.597800+0000","last_clean":"2024-09-17T09:08:09.597800+0000","last_became_active":"2024-09-17T09:08:00.410647+0000","last_became_peered":"2024-09-17T09:08:00.410647+0000","last_unstale":"2024-09-17T09:08:09.597800+0000","last_undegraded":"2024-09-17T09:08:09.597800+0000","last_fullsized":"2024-09-17T09:08:09.597800+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:35.637104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":35,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583449+0000","last_change":"2024-09-17T09:07:56.257146+0000","last_active":"2024-09-17T09:08:08.583449+0000","last_peered":"2024-09-17T09:08:08.583449+0000","last_clean":"2024-09-17T09:08:08.583449+0000","last_became_active":"2024-09-17T09:07:56.256714+0000","last_became_peered":"2024-09-17T09:07:56.256714+0000","last_unstale":"2024-09-17T09:08:08.583449+0000","last_undegraded":"2024-09-17T09:08:08.583449+0000","last_fullsized":"2024-09-17T09:08:08.583449+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:49.062390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584148+0000","last_change":"2024-09-17T09:07:54.252996+0000","last_active":"2024-09-17T09:08:08.584148+0000","last_peered":"2024-09-17T09:08:08.584148+0000","last_clean":"2024-09-17T09:08:08.584148+0000","last_became_active":"2024-09-17T09:07:54.252722+0000","last_became_peered":"2024-09-17T09:07:54.252722+0000","last_unstale":"2024-09-17T09:08:08.584148+0000","last_undegraded":"2024-09-17T09:08:08.584148+0000","last_fullsized":"2024-09-17T09:08:08.584148+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:42:45.445678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":103,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:16.027113+0000","last_change":"2024-09-17T09:07:58.267104+0000","last_active":"2024-09-17T09:09:16.027113+0000","last_peered":"2024-09-17T09:09:16.027113+0000","last_clean":"2024-09-17T09:09:16.027113+0000","last_became_active":"2024-09-17T09:07:54.248032+0000","last_became_peered":"2024-09-17T09:07:54.248032+0000","last_unstale":"2024-09-17T09:09:16.027113+0000","last_undegraded":"2024-09-17T09:09:16.027113+0000","last_fullsized":"2024-09-17T09:09:16.027113+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:09:30.903399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033981199999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583036+0000","last_change":"2024-09-17T09:07:58.278352+0000","last_active":"2024-09-17T09:08:08.583036+0000","last_peered":"2024-09-17T09:08:08.583036+0000","last_clean":"2024-09-17T09:08:08.583036+0000","last_became_active":"2024-09-17T09:07:58.278034+0000","last_became_peered":"2024-09-17T09:07:58.278034+0000","last_unstale":"2024-09-17T09:08:08.583036+0000","last_undegraded":"2024-09-17T09:08:08.583036+0000","last_fullsized":"2024-09-17T09:08:08.583036+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:43.714637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584001+0000","last_change":"2024-09-17T09:08:00.281783+0000","last_active":"2024-09-17T09:08:08.584001+0000","last_peered":"2024-09-17T09:08:08.584001+0000","last_clean":"2024-09-17T09:08:08.584001+0000","last_became_active":"2024-09-17T09:08:00.281462+0000","last_became_peered":"2024-09-17T09:08:00.281462+0000","last_unstale":"2024-09-17T09:08:08.584001+0000","last_undegraded":"2024-09-17T09:08:08.584001+0000","last_fullsized":"2024-09-17T09:08:08.584001+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:22.885975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"67'14","reported_seq":39,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547923+0000","last_change":"2024-09-17T09:07:56.250715+0000","last_active":"2024-09-17T09:08:09.547923+0000","last_peered":"2024-09-17T09:08:09.547923+0000","last_clean":"2024-09-17T09:08:09.547923+0000","last_became_active":"2024-09-17T09:07:56.250479+0000","last_became_peered":"2024-09-17T09:07:56.250479+0000","last_unstale":"2024-09-17T09:08:09.547923+0000","last_undegraded":"2024-09-17T09:08:09.547923+0000","last_fullsized":"2024-09-17T09:08:09.547923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:05.365570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598261+0000","last_change":"2024-09-17T09:07:54.251571+0000","last_active":"2024-09-17T09:08:09.598261+0000","last_peered":"2024-09-17T09:08:09.598261+0000","last_clean":"2024-09-17T09:08:09.598261+0000","last_became_active":"2024-09-17T09:07:54.251261+0000","last_became_peered":"2024-09-17T09:07:54.251261+0000","last_unstale":"2024-09-17T09:08:09.598261+0000","last_undegraded":"2024-09-17T09:08:09.598261+0000","last_fullsized":"2024-09-17T09:08:09.598261+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:02.409285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"69'128","reported_seq":265,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:08:33.416518+0000","last_change":"2024-09-17T09:07:44.430610+0000","last_active":"2024-09-17T09:08:33.416518+0000","last_peered":"2024-09-17T09:08:33.416518+0000","last_clean":"2024-09-17T09:08:33.416518+0000","last_became_active":"2024-09-17T09:07:44.430282+0000","last_became_peered":"2024-09-17T09:07:44.430282+0000","last_unstale":"2024-09-17T09:08:33.416518+0000","last_undegraded":"2024-09-17T09:08:33.416518+0000","last_fullsized":"2024-09-17T09:08:33.416518+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:05:43.688522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:05:43.688522+0000","last_clean_scrub_stamp":"2024-09-17T09:05:43.688522+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:38:07.261433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597465+0000","last_change":"2024-09-17T09:07:58.276429+0000","last_active":"2024-09-17T09:08:09.597465+0000","last_peered":"2024-09-17T09:08:09.597465+0000","last_clean":"2024-09-17T09:08:09.597465+0000","last_became_active":"2024-09-17T09:07:58.276143+0000","last_became_peered":"2024-09-17T09:07:58.276143+0000","last_unstale":"2024-09-17T09:08:09.597465+0000","last_undegraded":"2024-09-17T09:08:09.597465+0000","last_fullsized":"2024-09-17T09:08:09.597465+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:56:47.012931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582956+0000","last_change":"2024-09-17T09:08:00.282321+0000","last_active":"2024-09-17T09:08:08.582956+0000","last_peered":"2024-09-17T09:08:08.582956+0000","last_clean":"2024-09-17T09:08:08.582956+0000","last_became_active":"2024-09-17T09:08:00.282042+0000","last_became_peered":"2024-09-17T09:08:00.282042+0000","last_unstale":"2024-09-17T09:08:08.582956+0000","last_undegraded":"2024-09-17T09:08:08.582956+0000","last_fullsized":"2024-09-17T09:08:08.582956+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:13:06.700284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597591+0000","last_change":"2024-09-17T09:07:56.261402+0000","last_active":"2024-09-17T09:08:09.597591+0000","last_peered":"2024-09-17T09:08:09.597591+0000","last_clean":"2024-09-17T09:08:09.597591+0000","last_became_active":"2024-09-17T09:07:56.261120+0000","last_became_peered":"2024-09-17T09:07:56.261120+0000","last_unstale":"2024-09-17T09:08:09.597591+0000","last_undegraded":"2024-09-17T09:08:09.597591+0000","last_fullsized":"2024-09-17T09:08:09.597591+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:44.704514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597162+0000","last_change":"2024-09-17T09:07:54.247019+0000","last_active":"2024-09-17T09:08:09.597162+0000","last_peered":"2024-09-17T09:08:09.597162+0000","last_clean":"2024-09-17T09:08:09.597162+0000","last_became_active":"2024-09-17T09:07:54.246556+0000","last_became_peered":"2024-09-17T09:07:54.246556+0000","last_unstale":"2024-09-17T09:08:09.597162+0000","last_undegraded":"2024-09-17T09:08:09.597162+0000","last_fullsized":"2024-09-17T09:08:09.597162+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:39:10.724050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582885+0000","last_change":"2024-09-17T09:07:58.279814+0000","last_active":"2024-09-17T09:08:08.582885+0000","last_peered":"2024-09-17T09:08:08.582885+0000","last_clean":"2024-09-17T09:08:08.582885+0000","last_became_active":"2024-09-17T09:07:58.279498+0000","last_became_peered":"2024-09-17T09:07:58.279498+0000","last_unstale":"2024-09-17T09:08:08.582885+0000","last_undegraded":"2024-09-17T09:08:08.582885+0000","last_fullsized":"2024-09-17T09:08:08.582885+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:08:32.224061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597900+0000","last_change":"2024-09-17T09:08:00.411235+0000","last_active":"2024-09-17T09:08:09.597900+0000","last_peered":"2024-09-17T09:08:09.597900+0000","last_clean":"2024-09-17T09:08:09.597900+0000","last_became_active":"2024-09-17T09:08:00.410915+0000","last_became_peered":"2024-09-17T09:08:00.410915+0000","last_unstale":"2024-09-17T09:08:09.597900+0000","last_undegraded":"2024-09-17T09:08:09.597900+0000","last_fullsized":"2024-09-17T09:08:09.597900+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:57.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":45,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597624+0000","last_change":"2024-09-17T09:07:56.263357+0000","last_active":"2024-09-17T09:08:09.597624+0000","last_peered":"2024-09-17T09:08:09.597624+0000","last_clean":"2024-09-17T09:08:09.597624+0000","last_became_active":"2024-09-17T09:07:56.263029+0000","last_became_peered":"2024-09-17T09:07:56.263029+0000","last_unstale":"2024-09-17T09:08:09.597624+0000","last_undegraded":"2024-09-17T09:08:09.597624+0000","last_fullsized":"2024-09-17T09:08:09.597624+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.854310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547739+0000","last_change":"2024-09-17T09:07:54.251577+0000","last_active":"2024-09-17T09:08:09.547739+0000","last_peered":"2024-09-17T09:08:09.547739+0000","last_clean":"2024-09-17T09:08:09.547739+0000","last_became_active":"2024-09-17T09:07:54.251335+0000","last_became_peered":"2024-09-17T09:07:54.251335+0000","last_unstale":"2024-09-17T09:08:09.547739+0000","last_undegraded":"2024-09-17T09:08:09.547739+0000","last_fullsized":"2024-09-17T09:08:09.547739+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:31.606953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584078+0000","last_change":"2024-09-17T09:07:58.278665+0000","last_active":"2024-09-17T09:08:08.584078+0000","last_peered":"2024-09-17T09:08:08.584078+0000","last_clean":"2024-09-17T09:08:08.584078+0000","last_became_active":"2024-09-17T09:07:58.278335+0000","last_became_peered":"2024-09-17T09:07:58.278335+0000","last_unstale":"2024-09-17T09:08:08.584078+0000","last_undegraded":"2024-09-17T09:08:08.584078+0000","last_fullsized":"2024-09-17T09:08:08.584078+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:30.555707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597203+0000","last_change":"2024-09-17T09:08:00.271281+0000","last_active":"2024-09-17T09:08:09.597203+0000","last_peered":"2024-09-17T09:08:09.597203+0000","last_clean":"2024-09-17T09:08:09.597203+0000","last_became_active":"2024-09-17T09:08:00.270928+0000","last_became_peered":"2024-09-17T09:08:00.270928+0000","last_unstale":"2024-09-17T09:08:09.597203+0000","last_undegraded":"2024-09-17T09:08:09.597203+0000","last_fullsized":"2024-09-17T09:08:09.597203+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:24:12.214954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597049+0000","last_change":"2024-09-17T09:07:56.255100+0000","last_active":"2024-09-17T09:08:09.597049+0000","last_peered":"2024-09-17T09:08:09.597049+0000","last_clean":"2024-09-17T09:08:09.597049+0000","last_became_active":"2024-09-17T09:07:56.254843+0000","last_became_peered":"2024-09-17T09:07:56.254843+0000","last_unstale":"2024-09-17T09:08:09.597049+0000","last_undegraded":"2024-09-17T09:08:09.597049+0000","last_fullsized":"2024-09-17T09:08:09.597049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:10.512387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598558+0000","last_change":"2024-09-17T09:07:54.249030+0000","last_active":"2024-09-17T09:08:09.598558+0000","last_peered":"2024-09-17T09:08:09.598558+0000","last_clean":"2024-09-17T09:08:09.598558+0000","last_became_active":"2024-09-17T09:07:54.248698+0000","last_became_peered":"2024-09-17T09:07:54.248698+0000","last_unstale":"2024-09-17T09:08:09.598558+0000","last_undegraded":"2024-09-17T09:08:09.598558+0000","last_fullsized":"2024-09-17T09:08:09.598558+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:56:07.518482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":39,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:07.704687+0000","last_change":"2024-09-17T09:07:58.273256+0000","last_active":"2024-09-17T09:09:07.704687+0000","last_peered":"2024-09-17T09:09:07.704687+0000","last_clean":"2024-09-17T09:09:07.704687+0000","last_became_active":"2024-09-17T09:07:58.272951+0000","last_became_peered":"2024-09-17T09:07:58.272951+0000","last_unstale":"2024-09-17T09:09:07.704687+0000","last_undegraded":"2024-09-17T09:09:07.704687+0000","last_fullsized":"2024-09-17T09:09:07.704687+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:26:06.792961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596973+0000","last_change":"2024-09-17T09:08:00.278487+0000","last_active":"2024-09-17T09:08:09.596973+0000","last_peered":"2024-09-17T09:08:09.596973+0000","last_clean":"2024-09-17T09:08:09.596973+0000","last_became_active":"2024-09-17T09:08:00.278151+0000","last_became_peered":"2024-09-17T09:08:00.278151+0000","last_unstale":"2024-09-17T09:08:09.596973+0000","last_undegraded":"2024-09-17T09:08:09.596973+0000","last_fullsized":"2024-09-17T09:08:09.596973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:04.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584484+0000","last_change":"2024-09-17T09:07:56.259436+0000","last_active":"2024-09-17T09:08:08.584484+0000","last_peered":"2024-09-17T09:08:08.584484+0000","last_clean":"2024-09-17T09:08:08.584484+0000","last_became_active":"2024-09-17T09:07:56.258702+0000","last_became_peered":"2024-09-17T09:07:56.258702+0000","last_unstale":"2024-09-17T09:08:08.584484+0000","last_undegraded":"2024-09-17T09:08:08.584484+0000","last_fullsized":"2024-09-17T09:08:08.584484+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:31.375700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582474+0000","last_change":"2024-09-17T09:07:54.254581+0000","last_active":"2024-09-17T09:08:08.582474+0000","last_peered":"2024-09-17T09:08:08.582474+0000","last_clean":"2024-09-17T09:08:08.582474+0000","last_became_active":"2024-09-17T09:07:54.254270+0000","last_became_peered":"2024-09-17T09:07:54.254270+0000","last_unstale":"2024-09-17T09:08:08.582474+0000","last_undegraded":"2024-09-17T09:08:08.582474+0000","last_fullsized":"2024-09-17T09:08:08.582474+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:22.602874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597062+0000","last_change":"2024-09-17T09:07:58.269859+0000","last_active":"2024-09-17T09:08:09.597062+0000","last_peered":"2024-09-17T09:08:09.597062+0000","last_clean":"2024-09-17T09:08:09.597062+0000","last_became_active":"2024-09-17T09:07:58.269553+0000","last_became_peered":"2024-09-17T09:07:58.269553+0000","last_unstale":"2024-09-17T09:08:09.597062+0000","last_undegraded":"2024-09-17T09:08:09.597062+0000","last_fullsized":"2024-09-17T09:08:09.597062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:23:12.227060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.584074+0000","last_change":"2024-09-17T09:08:00.283656+0000","last_active":"2024-09-17T09:08:08.584074+0000","last_peered":"2024-09-17T09:08:08.584074+0000","last_clean":"2024-09-17T09:08:08.584074+0000","last_became_active":"2024-09-17T09:08:00.283326+0000","last_became_peered":"2024-09-17T09:08:00.283326+0000","last_unstale":"2024-09-17T09:08:08.584074+0000","last_undegraded":"2024-09-17T09:08:08.584074+0000","last_fullsized":"2024-09-17T09:08:08.584074+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:10.953066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":48,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547636+0000","last_change":"2024-09-17T09:07:56.256656+0000","last_active":"2024-09-17T09:08:09.547636+0000","last_peered":"2024-09-17T09:08:09.547636+0000","last_clean":"2024-09-17T09:08:09.547636+0000","last_became_active":"2024-09-17T09:07:56.256381+0000","last_became_peered":"2024-09-17T09:07:56.256381+0000","last_unstale":"2024-09-17T09:08:09.547636+0000","last_undegraded":"2024-09-17T09:08:09.547636+0000","last_fullsized":"2024-09-17T09:08:09.547636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:28:28.393034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598035+0000","last_change":"2024-09-17T09:07:54.253742+0000","last_active":"2024-09-17T09:08:09.598035+0000","last_peered":"2024-09-17T09:08:09.598035+0000","last_clean":"2024-09-17T09:08:09.598035+0000","last_became_active":"2024-09-17T09:07:54.253250+0000","last_became_peered":"2024-09-17T09:07:54.253250+0000","last_unstale":"2024-09-17T09:08:09.598035+0000","last_undegraded":"2024-09-17T09:08:09.598035+0000","last_fullsized":"2024-09-17T09:08:09.598035+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:16.198331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597059+0000","last_change":"2024-09-17T09:07:58.276592+0000","last_active":"2024-09-17T09:08:09.597059+0000","last_peered":"2024-09-17T09:08:09.597059+0000","last_clean":"2024-09-17T09:08:09.597059+0000","last_became_active":"2024-09-17T09:07:58.276316+0000","last_became_peered":"2024-09-17T09:07:58.276316+0000","last_unstale":"2024-09-17T09:08:09.597059+0000","last_undegraded":"2024-09-17T09:08:09.597059+0000","last_fullsized":"2024-09-17T09:08:09.597059+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:47.085654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596932+0000","last_change":"2024-09-17T09:08:00.284383+0000","last_active":"2024-09-17T09:08:09.596932+0000","last_peered":"2024-09-17T09:08:09.596932+0000","last_clean":"2024-09-17T09:08:09.596932+0000","last_became_active":"2024-09-17T09:08:00.284154+0000","last_became_peered":"2024-09-17T09:08:00.284154+0000","last_unstale":"2024-09-17T09:08:09.596932+0000","last_undegraded":"2024-09-17T09:08:09.596932+0000","last_fullsized":"2024-09-17T09:08:09.596932+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:17.442555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":40,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597030+0000","last_change":"2024-09-17T09:07:56.262185+0000","last_active":"2024-09-17T09:08:09.597030+0000","last_peered":"2024-09-17T09:08:09.597030+0000","last_clean":"2024-09-17T09:08:09.597030+0000","last_became_active":"2024-09-17T09:07:56.261514+0000","last_became_peered":"2024-09-17T09:07:56.261514+0000","last_unstale":"2024-09-17T09:08:09.597030+0000","last_undegraded":"2024-09-17T09:08:09.597030+0000","last_fullsized":"2024-09-17T09:08:09.597030+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:00:07.002282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598160+0000","last_change":"2024-09-17T09:07:54.251430+0000","last_active":"2024-09-17T09:08:09.598160+0000","last_peered":"2024-09-17T09:08:09.598160+0000","last_clean":"2024-09-17T09:08:09.598160+0000","last_became_active":"2024-09-17T09:07:54.251145+0000","last_became_peered":"2024-09-17T09:07:54.251145+0000","last_unstale":"2024-09-17T09:08:09.598160+0000","last_undegraded":"2024-09-17T09:08:09.598160+0000","last_fullsized":"2024-09-17T09:08:09.598160+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:01.036138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596983+0000","last_change":"2024-09-17T09:07:58.272732+0000","last_active":"2024-09-17T09:08:09.596983+0000","last_peered":"2024-09-17T09:08:09.596983+0000","last_clean":"2024-09-17T09:08:09.596983+0000","last_became_active":"2024-09-17T09:07:58.272334+0000","last_became_peered":"2024-09-17T09:07:58.272334+0000","last_unstale":"2024-09-17T09:08:09.596983+0000","last_undegraded":"2024-09-17T09:08:09.596983+0000","last_fullsized":"2024-09-17T09:08:09.596983+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:55:17.168854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597782+0000","last_change":"2024-09-17T09:08:00.279405+0000","last_active":"2024-09-17T09:08:09.597782+0000","last_peered":"2024-09-17T09:08:09.597782+0000","last_clean":"2024-09-17T09:08:09.597782+0000","last_became_active":"2024-09-17T09:08:00.279057+0000","last_became_peered":"2024-09-17T09:08:00.279057+0000","last_unstale":"2024-09-17T09:08:09.597782+0000","last_undegraded":"2024-09-17T09:08:09.597782+0000","last_fullsized":"2024-09-17T09:08:09.597782+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:51.833526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582074+0000","last_change":"2024-09-17T09:07:56.249475+0000","last_active":"2024-09-17T09:08:08.582074+0000","last_peered":"2024-09-17T09:08:08.582074+0000","last_clean":"2024-09-17T09:08:08.582074+0000","last_became_active":"2024-09-17T09:07:56.248230+0000","last_became_peered":"2024-09-17T09:07:56.248230+0000","last_unstale":"2024-09-17T09:08:08.582074+0000","last_undegraded":"2024-09-17T09:08:08.582074+0000","last_fullsized":"2024-09-17T09:08:08.582074+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:43.442867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":26,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597854+0000","last_change":"2024-09-17T09:07:54.252748+0000","last_active":"2024-09-17T09:08:09.597854+0000","last_peered":"2024-09-17T09:08:09.597854+0000","last_clean":"2024-09-17T09:08:09.597854+0000","last_became_active":"2024-09-17T09:07:54.249944+0000","last_became_peered":"2024-09-17T09:07:54.249944+0000","last_unstale":"2024-09-17T09:08:09.597854+0000","last_undegraded":"2024-09-17T09:08:09.597854+0000","last_fullsized":"2024-09-17T09:08:09.597854+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:16.653591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":39,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:07.704939+0000","last_change":"2024-09-17T09:07:58.274512+0000","last_active":"2024-09-17T09:09:07.704939+0000","last_peered":"2024-09-17T09:09:07.704939+0000","last_clean":"2024-09-17T09:09:07.704939+0000","last_became_active":"2024-09-17T09:07:58.274254+0000","last_became_peered":"2024-09-17T09:07:58.274254+0000","last_unstale":"2024-09-17T09:09:07.704939+0000","last_undegraded":"2024-09-17T09:09:07.704939+0000","last_fullsized":"2024-09-17T09:09:07.704939+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:15:32.353987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582242+0000","last_change":"2024-09-17T09:08:00.410948+0000","last_active":"2024-09-17T09:08:08.582242+0000","last_peered":"2024-09-17T09:08:08.582242+0000","last_clean":"2024-09-17T09:08:08.582242+0000","last_became_active":"2024-09-17T09:08:00.410659+0000","last_became_peered":"2024-09-17T09:08:00.410659+0000","last_unstale":"2024-09-17T09:08:08.582242+0000","last_undegraded":"2024-09-17T09:08:08.582242+0000","last_fullsized":"2024-09-17T09:08:08.582242+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:15.009371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547550+0000","last_change":"2024-09-17T09:07:54.249989+0000","last_active":"2024-09-17T09:08:09.547550+0000","last_peered":"2024-09-17T09:08:09.547550+0000","last_clean":"2024-09-17T09:08:09.547550+0000","last_became_active":"2024-09-17T09:07:54.249717+0000","last_became_peered":"2024-09-17T09:07:54.249717+0000","last_unstale":"2024-09-17T09:08:09.547550+0000","last_undegraded":"2024-09-17T09:08:09.547550+0000","last_fullsized":"2024-09-17T09:08:09.547550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:58:26.320619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":26,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598770+0000","last_change":"2024-09-17T09:07:56.259126+0000","last_active":"2024-09-17T09:08:09.598770+0000","last_peered":"2024-09-17T09:08:09.598770+0000","last_clean":"2024-09-17T09:08:09.598770+0000","last_became_active":"2024-09-17T09:07:56.258897+0000","last_became_peered":"2024-09-17T09:07:56.258897+0000","last_unstale":"2024-09-17T09:08:09.598770+0000","last_undegraded":"2024-09-17T09:08:09.598770+0000","last_fullsized":"2024-09-17T09:08:09.598770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:39.212779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582799+0000","last_change":"2024-09-17T09:07:58.272347+0000","last_active":"2024-09-17T09:08:08.582799+0000","last_peered":"2024-09-17T09:08:08.582799+0000","last_clean":"2024-09-17T09:08:08.582799+0000","last_became_active":"2024-09-17T09:07:58.272031+0000","last_became_peered":"2024-09-17T09:07:58.272031+0000","last_unstale":"2024-09-17T09:08:08.582799+0000","last_undegraded":"2024-09-17T09:08:08.582799+0000","last_fullsized":"2024-09-17T09:08:08.582799+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:15.727684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597722+0000","last_change":"2024-09-17T09:08:00.412017+0000","last_active":"2024-09-17T09:08:09.597722+0000","last_peered":"2024-09-17T09:08:09.597722+0000","last_clean":"2024-09-17T09:08:09.597722+0000","last_became_active":"2024-09-17T09:08:00.411721+0000","last_became_peered":"2024-09-17T09:08:00.411721+0000","last_unstale":"2024-09-17T09:08:09.597722+0000","last_undegraded":"2024-09-17T09:08:09.597722+0000","last_fullsized":"2024-09-17T09:08:09.597722+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:21.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583536+0000","last_change":"2024-09-17T09:07:56.256979+0000","last_active":"2024-09-17T09:08:08.583536+0000","last_peered":"2024-09-17T09:08:08.583536+0000","last_clean":"2024-09-17T09:08:08.583536+0000","last_became_active":"2024-09-17T09:07:56.256324+0000","last_became_peered":"2024-09-17T09:07:56.256324+0000","last_unstale":"2024-09-17T09:08:08.583536+0000","last_undegraded":"2024-09-17T09:08:08.583536+0000","last_fullsized":"2024-09-17T09:08:08.583536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:53.378196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597429+0000","last_change":"2024-09-17T09:07:54.250742+0000","last_active":"2024-09-17T09:08:09.597429+0000","last_peered":"2024-09-17T09:08:09.597429+0000","last_clean":"2024-09-17T09:08:09.597429+0000","last_became_active":"2024-09-17T09:07:54.248384+0000","last_became_peered":"2024-09-17T09:07:54.248384+0000","last_unstale":"2024-09-17T09:08:09.597429+0000","last_undegraded":"2024-09-17T09:08:09.597429+0000","last_fullsized":"2024-09-17T09:08:09.597429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:16.872233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598688+0000","last_change":"2024-09-17T09:07:58.275761+0000","last_active":"2024-09-17T09:08:09.598688+0000","last_peered":"2024-09-17T09:08:09.598688+0000","last_clean":"2024-09-17T09:08:09.598688+0000","last_became_active":"2024-09-17T09:07:58.275481+0000","last_became_peered":"2024-09-17T09:07:58.275481+0000","last_unstale":"2024-09-17T09:08:09.598688+0000","last_undegraded":"2024-09-17T09:08:09.598688+0000","last_fullsized":"2024-09-17T09:08:09.598688+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:14.444429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597295+0000","last_change":"2024-09-17T09:08:00.275968+0000","last_active":"2024-09-17T09:08:09.597295+0000","last_peered":"2024-09-17T09:08:09.597295+0000","last_clean":"2024-09-17T09:08:09.597295+0000","last_became_active":"2024-09-17T09:08:00.275656+0000","last_became_peered":"2024-09-17T09:08:00.275656+0000","last_unstale":"2024-09-17T09:08:09.597295+0000","last_undegraded":"2024-09-17T09:08:09.597295+0000","last_fullsized":"2024-09-17T09:08:09.597295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:49:51.821355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582710+0000","last_change":"2024-09-17T09:07:56.254319+0000","last_active":"2024-09-17T09:08:08.582710+0000","last_peered":"2024-09-17T09:08:08.582710+0000","last_clean":"2024-09-17T09:08:08.582710+0000","last_became_active":"2024-09-17T09:07:56.254008+0000","last_became_peered":"2024-09-17T09:07:56.254008+0000","last_unstale":"2024-09-17T09:08:08.582710+0000","last_undegraded":"2024-09-17T09:08:08.582710+0000","last_fullsized":"2024-09-17T09:08:08.582710+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:45.719498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583909+0000","last_change":"2024-09-17T09:07:54.250064+0000","last_active":"2024-09-17T09:08:08.583909+0000","last_peered":"2024-09-17T09:08:08.583909+0000","last_clean":"2024-09-17T09:08:08.583909+0000","last_became_active":"2024-09-17T09:07:54.249746+0000","last_became_peered":"2024-09-17T09:07:54.249746+0000","last_unstale":"2024-09-17T09:08:08.583909+0000","last_undegraded":"2024-09-17T09:08:08.583909+0000","last_fullsized":"2024-09-17T09:08:08.583909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:00:19.112070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":39,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:07.704914+0000","last_change":"2024-09-17T09:07:58.272858+0000","last_active":"2024-09-17T09:09:07.704914+0000","last_peered":"2024-09-17T09:09:07.704914+0000","last_clean":"2024-09-17T09:09:07.704914+0000","last_became_active":"2024-09-17T09:07:58.272525+0000","last_became_peered":"2024-09-17T09:07:58.272525+0000","last_unstale":"2024-09-17T09:09:07.704914+0000","last_undegraded":"2024-09-17T09:09:07.704914+0000","last_fullsized":"2024-09-17T09:09:07.704914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:24:12.623502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597860+0000","last_change":"2024-09-17T09:08:00.273280+0000","last_active":"2024-09-17T09:08:09.597860+0000","last_peered":"2024-09-17T09:08:09.597860+0000","last_clean":"2024-09-17T09:08:09.597860+0000","last_became_active":"2024-09-17T09:08:00.272978+0000","last_became_peered":"2024-09-17T09:08:00.272978+0000","last_unstale":"2024-09-17T09:08:09.597860+0000","last_undegraded":"2024-09-17T09:08:09.597860+0000","last_fullsized":"2024-09-17T09:08:09.597860+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:16.164898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598634+0000","last_change":"2024-09-17T09:07:56.257915+0000","last_active":"2024-09-17T09:08:09.598634+0000","last_peered":"2024-09-17T09:08:09.598634+0000","last_clean":"2024-09-17T09:08:09.598634+0000","last_became_active":"2024-09-17T09:07:56.257657+0000","last_became_peered":"2024-09-17T09:07:56.257657+0000","last_unstale":"2024-09-17T09:08:09.598634+0000","last_undegraded":"2024-09-17T09:08:09.598634+0000","last_fullsized":"2024-09-17T09:08:09.598634+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:49.233892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597612+0000","last_change":"2024-09-17T09:07:54.252390+0000","last_active":"2024-09-17T09:08:09.597612+0000","last_peered":"2024-09-17T09:08:09.597612+0000","last_clean":"2024-09-17T09:08:09.597612+0000","last_became_active":"2024-09-17T09:07:54.251943+0000","last_became_peered":"2024-09-17T09:07:54.251943+0000","last_unstale":"2024-09-17T09:08:09.597612+0000","last_undegraded":"2024-09-17T09:08:09.597612+0000","last_fullsized":"2024-09-17T09:08:09.597612+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:10.164114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":39,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-17T09:09:07.704945+0000","last_change":"2024-09-17T09:07:58.278470+0000","last_active":"2024-09-17T09:09:07.704945+0000","last_peered":"2024-09-17T09:09:07.704945+0000","last_clean":"2024-09-17T09:09:07.704945+0000","last_became_active":"2024-09-17T09:07:58.278208+0000","last_became_peered":"2024-09-17T09:07:58.278208+0000","last_unstale":"2024-09-17T09:09:07.704945+0000","last_undegraded":"2024-09-17T09:09:07.704945+0000","last_fullsized":"2024-09-17T09:09:07.704945+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:12.960474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583079+0000","last_change":"2024-09-17T09:08:00.282545+0000","last_active":"2024-09-17T09:08:08.583079+0000","last_peered":"2024-09-17T09:08:08.583079+0000","last_clean":"2024-09-17T09:08:08.583079+0000","last_became_active":"2024-09-17T09:08:00.282264+0000","last_became_peered":"2024-09-17T09:08:00.282264+0000","last_unstale":"2024-09-17T09:08:08.583079+0000","last_undegraded":"2024-09-17T09:08:08.583079+0000","last_fullsized":"2024-09-17T09:08:08.583079+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:15.713272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597083+0000","last_change":"2024-09-17T09:07:56.261507+0000","last_active":"2024-09-17T09:08:09.597083+0000","last_peered":"2024-09-17T09:08:09.597083+0000","last_clean":"2024-09-17T09:08:09.597083+0000","last_became_active":"2024-09-17T09:07:56.261188+0000","last_became_peered":"2024-09-17T09:07:56.261188+0000","last_unstale":"2024-09-17T09:08:09.597083+0000","last_undegraded":"2024-09-17T09:08:09.597083+0000","last_fullsized":"2024-09-17T09:08:09.597083+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:59.493678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597800+0000","last_change":"2024-09-17T09:07:54.251442+0000","last_active":"2024-09-17T09:08:09.597800+0000","last_peered":"2024-09-17T09:08:09.597800+0000","last_clean":"2024-09-17T09:08:09.597800+0000","last_became_active":"2024-09-17T09:07:54.251172+0000","last_became_peered":"2024-09-17T09:07:54.251172+0000","last_unstale":"2024-09-17T09:08:09.597800+0000","last_undegraded":"2024-09-17T09:08:09.597800+0000","last_fullsized":"2024-09-17T09:08:09.597800+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:31.284871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582936+0000","last_change":"2024-09-17T09:07:58.270640+0000","last_active":"2024-09-17T09:08:08.582936+0000","last_peered":"2024-09-17T09:08:08.582936+0000","last_clean":"2024-09-17T09:08:08.582936+0000","last_became_active":"2024-09-17T09:07:58.270338+0000","last_became_peered":"2024-09-17T09:07:58.270338+0000","last_unstale":"2024-09-17T09:08:08.582936+0000","last_undegraded":"2024-09-17T09:08:08.582936+0000","last_fullsized":"2024-09-17T09:08:08.582936+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:28:10.388495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597220+0000","last_change":"2024-09-17T09:08:00.278406+0000","last_active":"2024-09-17T09:08:09.597220+0000","last_peered":"2024-09-17T09:08:09.597220+0000","last_clean":"2024-09-17T09:08:09.597220+0000","last_became_active":"2024-09-17T09:08:00.278093+0000","last_became_peered":"2024-09-17T09:08:00.278093+0000","last_unstale":"2024-09-17T09:08:09.597220+0000","last_undegraded":"2024-09-17T09:08:09.597220+0000","last_fullsized":"2024-09-17T09:08:09.597220+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:38.162549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583201+0000","last_change":"2024-09-17T09:07:56.263088+0000","last_active":"2024-09-17T09:08:08.583201+0000","last_peered":"2024-09-17T09:08:08.583201+0000","last_clean":"2024-09-17T09:08:08.583201+0000","last_became_active":"2024-09-17T09:07:56.262820+0000","last_became_peered":"2024-09-17T09:07:56.262820+0000","last_unstale":"2024-09-17T09:08:08.583201+0000","last_undegraded":"2024-09-17T09:08:08.583201+0000","last_fullsized":"2024-09-17T09:08:08.583201+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:15:45.983694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597921+0000","last_change":"2024-09-17T09:07:54.251865+0000","last_active":"2024-09-17T09:08:09.597921+0000","last_peered":"2024-09-17T09:08:09.597921+0000","last_clean":"2024-09-17T09:08:09.597921+0000","last_became_active":"2024-09-17T09:07:54.251544+0000","last_became_peered":"2024-09-17T09:07:54.251544+0000","last_unstale":"2024-09-17T09:08:09.597921+0000","last_undegraded":"2024-09-17T09:08:09.597921+0000","last_fullsized":"2024-09-17T09:08:09.597921+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:34.212625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597877+0000","last_change":"2024-09-17T09:07:58.277185+0000","last_active":"2024-09-17T09:08:09.597877+0000","last_peered":"2024-09-17T09:08:09.597877+0000","last_clean":"2024-09-17T09:08:09.597877+0000","last_became_active":"2024-09-17T09:07:58.276904+0000","last_became_peered":"2024-09-17T09:07:58.276904+0000","last_unstale":"2024-09-17T09:08:09.597877+0000","last_undegraded":"2024-09-17T09:08:09.597877+0000","last_fullsized":"2024-09-17T09:08:09.597877+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:46.991179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582480+0000","last_change":"2024-09-17T09:08:00.280989+0000","last_active":"2024-09-17T09:08:08.582480+0000","last_peered":"2024-09-17T09:08:08.582480+0000","last_clean":"2024-09-17T09:08:08.582480+0000","last_became_active":"2024-09-17T09:08:00.280680+0000","last_became_peered":"2024-09-17T09:08:00.280680+0000","last_unstale":"2024-09-17T09:08:08.582480+0000","last_undegraded":"2024-09-17T09:08:08.582480+0000","last_fullsized":"2024-09-17T09:08:08.582480+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:13:28.443695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.597365+0000","last_change":"2024-09-17T09:07:56.247428+0000","last_active":"2024-09-17T09:08:09.597365+0000","last_peered":"2024-09-17T09:08:09.597365+0000","last_clean":"2024-09-17T09:08:09.597365+0000","last_became_active":"2024-09-17T09:07:56.247086+0000","last_became_peered":"2024-09-17T09:07:56.247086+0000","last_unstale":"2024-09-17T09:08:09.597365+0000","last_undegraded":"2024-09-17T09:08:09.597365+0000","last_fullsized":"2024-09-17T09:08:09.597365+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:10.997594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.582807+0000","last_change":"2024-09-17T09:07:54.246150+0000","last_active":"2024-09-17T09:08:08.582807+0000","last_peered":"2024-09-17T09:08:08.582807+0000","last_clean":"2024-09-17T09:08:08.582807+0000","last_became_active":"2024-09-17T09:07:54.245853+0000","last_became_peered":"2024-09-17T09:07:54.245853+0000","last_unstale":"2024-09-17T09:08:08.582807+0000","last_undegraded":"2024-09-17T09:08:08.582807+0000","last_fullsized":"2024-09-17T09:08:08.582807+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:43.443395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596638+0000","last_change":"2024-09-17T09:07:58.275230+0000","last_active":"2024-09-17T09:08:09.596638+0000","last_peered":"2024-09-17T09:08:09.596638+0000","last_clean":"2024-09-17T09:08:09.596638+0000","last_became_active":"2024-09-17T09:07:58.274829+0000","last_became_peered":"2024-09-17T09:07:58.274829+0000","last_unstale":"2024-09-17T09:08:09.596638+0000","last_undegraded":"2024-09-17T09:08:09.596638+0000","last_fullsized":"2024-09-17T09:08:09.596638+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:43.808216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598059+0000","last_change":"2024-09-17T09:08:00.413063+0000","last_active":"2024-09-17T09:08:09.598059+0000","last_peered":"2024-09-17T09:08:09.598059+0000","last_clean":"2024-09-17T09:08:09.598059+0000","last_became_active":"2024-09-17T09:08:00.412540+0000","last_became_peered":"2024-09-17T09:08:00.412540+0000","last_unstale":"2024-09-17T09:08:09.598059+0000","last_undegraded":"2024-09-17T09:08:09.598059+0000","last_fullsized":"2024-09-17T09:08:09.598059+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:42.269188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598072+0000","last_change":"2024-09-17T09:07:56.259721+0000","last_active":"2024-09-17T09:08:09.598072+0000","last_peered":"2024-09-17T09:08:09.598072+0000","last_clean":"2024-09-17T09:08:09.598072+0000","last_became_active":"2024-09-17T09:07:56.259478+0000","last_became_peered":"2024-09-17T09:07:56.259478+0000","last_unstale":"2024-09-17T09:08:09.598072+0000","last_undegraded":"2024-09-17T09:08:09.598072+0000","last_fullsized":"2024-09-17T09:08:09.598072+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:15:08.191052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583617+0000","last_change":"2024-09-17T09:07:54.247538+0000","last_active":"2024-09-17T09:08:08.583617+0000","last_peered":"2024-09-17T09:08:08.583617+0000","last_clean":"2024-09-17T09:08:08.583617+0000","last_became_active":"2024-09-17T09:07:54.247258+0000","last_became_peered":"2024-09-17T09:07:54.247258+0000","last_unstale":"2024-09-17T09:08:08.583617+0000","last_undegraded":"2024-09-17T09:08:08.583617+0000","last_fullsized":"2024-09-17T09:08:08.583617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:26.043135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.547539+0000","last_change":"2024-09-17T09:07:58.275169+0000","last_active":"2024-09-17T09:08:09.547539+0000","last_peered":"2024-09-17T09:08:09.547539+0000","last_clean":"2024-09-17T09:08:09.547539+0000","last_became_active":"2024-09-17T09:07:58.274877+0000","last_became_peered":"2024-09-17T09:07:58.274877+0000","last_unstale":"2024-09-17T09:08:09.547539+0000","last_undegraded":"2024-09-17T09:08:09.547539+0000","last_fullsized":"2024-09-17T09:08:09.547539+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:44.596599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.596767+0000","last_change":"2024-09-17T09:08:00.277968+0000","last_active":"2024-09-17T09:08:09.596767+0000","last_peered":"2024-09-17T09:08:09.596767+0000","last_clean":"2024-09-17T09:08:09.596767+0000","last_became_active":"2024-09-17T09:08:00.277594+0000","last_became_peered":"2024-09-17T09:08:00.277594+0000","last_unstale":"2024-09-17T09:08:09.596767+0000","last_undegraded":"2024-09-17T09:08:09.596767+0000","last_fullsized":"2024-09-17T09:08:09.596767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:28:27.846574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583525+0000","last_change":"2024-09-17T09:08:00.276935+0000","last_active":"2024-09-17T09:08:08.583525+0000","last_peered":"2024-09-17T09:08:08.583525+0000","last_clean":"2024-09-17T09:08:08.583525+0000","last_became_active":"2024-09-17T09:08:00.276429+0000","last_became_peered":"2024-09-17T09:08:00.276429+0000","last_unstale":"2024-09-17T09:08:08.583525+0000","last_undegraded":"2024-09-17T09:08:08.583525+0000","last_fullsized":"2024-09-17T09:08:08.583525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:59.254812+0000","last_clean_scrub_stamp":"2024-09-17T09:07:59.254812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:37.307114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.598265+0000","last_change":"2024-09-17T09:07:54.255587+0000","last_active":"2024-09-17T09:08:09.598265+0000","last_peered":"2024-09-17T09:08:09.598265+0000","last_clean":"2024-09-17T09:08:09.598265+0000","last_became_active":"2024-09-17T09:07:54.255131+0000","last_became_peered":"2024-09-17T09:07:54.255131+0000","last_unstale":"2024-09-17T09:08:09.598265+0000","last_undegraded":"2024-09-17T09:08:09.598265+0000","last_fullsized":"2024-09-17T09:08:09.598265+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:53.229281+0000","last_clean_scrub_stamp":"2024-09-17T09:07:53.229281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:28:56.681196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:09.548044+0000","last_change":"2024-09-17T09:07:56.256748+0000","last_active":"2024-09-17T09:08:09.548044+0000","last_peered":"2024-09-17T09:08:09.548044+0000","last_clean":"2024-09-17T09:08:09.548044+0000","last_became_active":"2024-09-17T09:07:56.256443+0000","last_became_peered":"2024-09-17T09:07:56.256443+0000","last_unstale":"2024-09-17T09:08:09.548044+0000","last_undegraded":"2024-09-17T09:08:09.548044+0000","last_fullsized":"2024-09-17T09:08:09.548044+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:55.235857+0000","last_clean_scrub_stamp":"2024-09-17T09:07:55.235857+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:31:43.401986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T09:08:08.583173+0000","last_change":"2024-09-17T09:07:58.276149+0000","last_active":"2024-09-17T09:08:08.583173+0000","last_peered":"2024-09-17T09:08:08.583173+0000","last_clean":"2024-09-17T09:08:08.583173+0000","last_became_active":"2024-09-17T09:07:58.275661+0000","last_became_peered":"2024-09-17T09:07:58.275661+0000","last_unstale":"2024-09-17T09:08:08.583173+0000","last_undegraded":"2024-09-17T09:08:08.583173+0000","last_fullsized":"2024-09-17T09:08:08.583173+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:07:57.248060+0000","last_clean_scrub_stamp":"2024-09-17T09:07:57.248060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:47.415050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32356,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712796,"statfs":{"total":95995035648,"available":95961903104,"internally_reserved":0,"allocated":1273856,"data_stored":855000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.60299999999999998}]},{"osd":1,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.49399999999999999}]},{"osd":2,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.504}]},{"osd":3,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.61899999999999999}]},{"osd":4,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.35599999999999998}]},{"osd":5,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.42299999999999999}]},{"osd":6,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.442},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.40400000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28236,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716916,"statfs":{"total":95995035648,"available":95966121984,"internally_reserved":0,"allocated":1249280,"data_stored":853529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.47299999999999998}]},{"osd":1,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.42299999999999999}]},{"osd":2,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.442},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.41299999999999998}]},{"osd":3,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.433},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.45100000000000001}]},{"osd":4,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.36699999999999999}]},{"osd":5,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.35599999999999998}]},{"osd":7,"last update":"Tue Sep 17 09:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.46300000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31768,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713384,"statfs":{"total":95995035648,"available":95962505216,"internally_reserved":0,"allocated":667648,"data_stored":263830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64700000000000002,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.498},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.58099999999999996}]},{"osd":1,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71599999999999997,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72599999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.75700000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70299999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.45600000000000002}]},{"osd":3,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.749,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51600000000000001,"15min":0.51100000000000001},"min":{"1min":0.435,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74199999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46899999999999997}]},{"osd":4,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.32200000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59999999999999998,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.38700000000000001,"15min":0.38200000000000001},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.67100000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.372}]},{"osd":6,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.34000000000000002}]},{"osd":7,"last update":"Tue Sep 17 09:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.40699999999999997}]}]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":692224,"data_stored":265312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52000000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61199999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53700000000000003,"15min":0.54700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64100000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.39600000000000002}]},{"osd":1,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69599999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67700000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.45400000000000001}]},{"osd":2,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64800000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.65600000000000003,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.48299999999999998}]},{"osd":3,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68200000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.55000000000000004,"15min":0.55700000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67400000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.40799999999999997}]},{"osd":5,"last update":"Tue Sep 17 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.34599999999999997,"15min":0.34000000000000002},"min":{"1min":0.30599999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.35599999999999998,"15min":0.34899999999999998},"min":{"1min":0.30599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.41499999999999998}]},{"osd":6,"last update":"Tue Sep 17 09:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.36299999999999999}]},{"osd":7,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.29899999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31800,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713352,"statfs":{"total":95995035648,"available":95962472448,"internally_reserved":0,"allocated":700416,"data_stored":265224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.379,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.48499999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.35099999999999998}]},{"osd":1,"last update":"Tue Sep 17 09:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41299999999999998,"15min":0.41899999999999998},"min":{"1min":0.29399999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.39800000000000002,"15min":0.41499999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.48599999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.32100000000000001}]},{"osd":2,"last update":"Tue Sep 17 09:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41699999999999998,"15min":0.42499999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.52700000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.40500000000000003,"15min":0.41099999999999998},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.57099999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.33900000000000002}]},{"osd":4,"last update":"Tue Sep 17 09:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.52200000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.505}]},{"osd":5,"last update":"Tue Sep 17 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38200000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52500000000000002,"15min":0.52000000000000002},"min":{"1min":0.41799999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.46100000000000002}]},{"osd":6,"last update":"Tue Sep 17 09:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.48499999999999999}]},{"osd":7,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.44},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.45000000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31748,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713404,"statfs":{"total":95995035648,"available":95962525696,"internally_reserved":0,"allocated":647168,"data_stored":263060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36499999999999999,"15min":0.34999999999999998},"min":{"1min":0.253,"5min":0.25,"15min":0.25},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.253},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.375,"15min":0.36299999999999999},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Tue Sep 17 09:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36299999999999999,"15min":0.35599999999999998},"min":{"1min":0.23599999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.41599999999999998,"15min":0.40300000000000002},"min":{"1min":0.28699999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.311}]},{"osd":3,"last update":"Tue Sep 17 09:09:12 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46300000000000002,"15min":0.46899999999999997},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.29299999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.33200000000000002}]},{"osd":4,"last update":"Tue Sep 17 09:08:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61699999999999999,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.376},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.52200000000000002}]},{"osd":5,"last update":"Tue Sep 17 09:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.46999999999999997,"5min":0.441,"15min":0.441},"max":{"1min":0.89600000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.433},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.60399999999999998,"15min":0.61099999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.44800000000000001}]},{"osd":6,"last update":"Tue Sep 17 09:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.54000000000000004}]},{"osd":7,"last update":"Tue Sep 17 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.39900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31812,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713340,"statfs":{"total":95995035648,"available":95962460160,"internally_reserved":0,"allocated":712704,"data_stored":265513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:08:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32400000000000001,"15min":0.30099999999999999},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.44900000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33200000000000002,"15min":0.313},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.495,"5min":0.496,"15min":0.496},"last":0.39600000000000002}]},{"osd":2,"last update":"Tue Sep 17 09:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36799999999999999,"15min":0.36099999999999999},"min":{"1min":0.23400000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.35499999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.502}]},{"osd":3,"last update":"Tue Sep 17 09:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.435,"15min":0.45500000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.57899999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.439,"15min":0.45700000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58899999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.42099999999999999}]},{"osd":4,"last update":"Tue Sep 17 09:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.60599999999999998}]},{"osd":5,"last update":"Tue Sep 17 09:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.629},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.51800000000000002}]},{"osd":6,"last update":"Tue Sep 17 09:08:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59299999999999997}]},{"osd":7,"last update":"Tue Sep 17 09:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.65100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32360,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712792,"statfs":{"total":95995035648,"available":95961899008,"internally_reserved":0,"allocated":1277952,"data_stored":855028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33600000000000002,"15min":0.32300000000000001},"min":{"1min":0.23799999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.53700000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.308},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.35099999999999998,"15min":0.33500000000000002},"min":{"1min":0.24199999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.51000000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.26000000000000001}]},{"osd":2,"last update":"Tue Sep 17 09:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51700000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.373,"15min":0.371},"min":{"1min":0.28899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.51000000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.34899999999999998}]},{"osd":3,"last update":"Tue Sep 17 09:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.41299999999999998,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52800000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.40400000000000003,"15min":0.41599999999999998},"min":{"1min":0.26300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.53600000000000003,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38500000000000001}]},{"osd":4,"last update":"Tue Sep 17 09:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54500000000000004,"15min":0.54200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.59399999999999997}]},{"osd":5,"last update":"Tue Sep 17 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60499999999999998}]},{"osd":6,"last update":"Tue Sep 17 09:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Sep 17 09:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.57099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T09:09:18.631 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-17T09:09:18.631 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T09:09:18.631 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-17T09:09:18.632 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-17T09:09:18.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:18 smithi173 ceph-mon[33421]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:18.831 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:09:18.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:18 smithi079 ceph-mon[29577]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:18.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:18 smithi079 ceph-mon[40839]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:19.590 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:19 smithi079 ceph-mon[29577]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:09:19.590 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:19 smithi079 ceph-mon[40839]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:09:19.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:19 smithi173 ceph-mon[33421]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:09:20.540 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-17T09:09:20.540 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T09:09:20.583 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:20 smithi173 ceph-mon[33421]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:20.839 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:20 smithi079 ceph-mon[29577]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:20.839 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:20 smithi079 ceph-mon[40839]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:21.111 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-17T09:09:21.111 INFO:tasks.cephadm:Setup complete, yielding 2024-09-17T09:09:21.111 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T09:09:21.121 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 721af936-74d3-11ef-bceb-c7b262605968 -- bash -c 'set -ex 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> do 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> echo "rotating key for $f" 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> K=$(ceph auth get-key $f) 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> NK="$K" 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> ceph orch daemon rotate-key $f 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> while [ "$K" == "$NK" ]; do 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> sleep 5 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> NK=$(ceph auth get-key $f) 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> done 2024-09-17T09:09:21.122 DEBUG:teuthology.orchestra.run.smithi079:> done 2024-09-17T09:09:21.123 DEBUG:teuthology.orchestra.run.smithi079:> ' 2024-09-17T09:09:21.319 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:09:21.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:21 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2864479865' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T09:09:21.701 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:21 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2864479865' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T09:09:21.702 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:21 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2864479865' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T09:09:22.420 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:22 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:22.375Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00278667s 2024-09-17T09:09:22.673 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:22 smithi079 ceph-mon[29577]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:22.673 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:22 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:22.673 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:22 smithi079 ceph-mon[40839]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:22.673 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:22 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:22.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:22 smithi173 ceph-mon[33421]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:22.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:22 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:22.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:09:22 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:09:22.882 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for osd.0 2024-09-17T09:09:22.882 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:09:22.882 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for osd.0' 2024-09-17T09:09:22.882 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:09:23.097 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:09:23.097 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:09:23.097 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key osd.0 2024-09-17T09:09:23.359 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key osd.0 on host 'smithi079' 2024-09-17T09:09:23.369 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:09:23.369 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:09:23.455 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 systemd[1]: Starting Ceph grafana.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:09:23.611 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:09:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:09:23.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1833833600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:23.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[40839]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:09:23.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[40839]: Schedule rotate-key daemon osd.0 2024-09-17T09:09:23.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:23.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:23.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[40839]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:23.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1833833600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:23.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[29577]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:09:23.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[29577]: Schedule rotate-key daemon osd.0 2024-09-17T09:09:23.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:23.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:23.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:23 smithi079 ceph-mon[29577]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:23.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1833833600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:23.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-mon[33421]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:09:23.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-mon[33421]: Schedule rotate-key daemon osd.0 2024-09-17T09:09:23.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:23.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:23.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-mon[33421]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:23.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 podman[66513]: 2024-09-17 09:09:23.465487707 +0000 UTC m=+0.015350346 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-17T09:09:23.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 podman[66513]: 2024-09-17 09:09:23.568096321 +0000 UTC m=+0.117958950 container create 8606b326879cff927394a6fa75e6e749bb60d37e2593635f13a5404adc0b1fb6 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.buildah.version=1.32.0, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, architecture=x86_64, com.redhat.component=ubi8-container, name=ubi8, distribution-scope=public, io.openshift.tags=base rhel8) 2024-09-17T09:09:24.072 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 podman[66513]: 2024-09-17 09:09:23.821015113 +0000 UTC m=+0.370877752 container init 8606b326879cff927394a6fa75e6e749bb60d37e2593635f13a5404adc0b1fb6 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a, vendor=Red Hat, Inc., maintainer=Paul Cuzner , vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, version=8.5, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897) 2024-09-17T09:09:24.072 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 podman[66513]: 2024-09-17 09:09:23.826285264 +0000 UTC m=+0.376147895 container start 8606b326879cff927394a6fa75e6e749bb60d37e2593635f13a5404adc0b1fb6 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , name=ubi8, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.openshift.expose-services=, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8) 2024-09-17T09:09:24.072 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 bash[66513]: 8606b326879cff927394a6fa75e6e749bb60d37e2593635f13a5404adc0b1fb6 2024-09-17T09:09:24.072 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 systemd[1]: Started Ceph grafana.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:09:24.072 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.97938194Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-17T09:09:24.072 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979597395Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-17T09:09:24.072 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979621194Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-17T09:09:24.072 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979634552Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-17T09:09:24.073 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979645003Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T09:09:24.073 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979655351Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T09:09:24.073 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979665759Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T09:09:24.073 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979675791Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T09:09:24.073 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979690117Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979700097Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979709652Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979719227Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.979729101Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=settings t=2024-09-17T09:09:23.97973926Z level=info msg="App mode production" 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=sqlstore t=2024-09-17T09:09:23.979821664Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=sqlstore t=2024-09-17T09:09:23.979847573Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.980686246Z level=info msg="Starting DB migrations" 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.980768267Z level=info msg="Executing migration" id="create migration_log table" 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.981279255Z level=info msg="Executing migration" id="create user table" 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.981723464Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.982517219Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-17T09:09:24.074 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.98348861Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.984103722Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.985072121Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.985832488Z level=info msg="Executing migration" id="create user table v2" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.986567664Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.986990092Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.987840095Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.988654618Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.989027709Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.98981764Z level=info msg="Executing migration" id="Update user table charset" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.990578575Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.991360909Z level=info msg="Executing migration" id="Add missing user data" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.99177164Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.992544466Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-17T09:09:24.075 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.993462922Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.994188762Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.995345918Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.99641136Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.997146334Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.998032358Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.998885183Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.999606393Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:23 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:23.999876258Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.000300809Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.000664844Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.001060505Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.001509802Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.002374553Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-17T09:09:24.076 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.003184108Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.004195675Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.005085528Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.007695873Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.010692975Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.013201107Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.016012838Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.016387787Z level=info msg="Executing migration" id="create star table" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.019035294Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.019504429Z level=info msg="Executing migration" id="create org table v1" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.021874217Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.024334323Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-17T09:09:24.077 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.027075952Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.029769686Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.032536989Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.032983692Z level=info msg="Executing migration" id="Update org table charset" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.035725015Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.038218151Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.041010452Z level=info msg="Executing migration" id="create dashboard table" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.041443781Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.044274037Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.046954832Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.049365167Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-17T09:09:24.078 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.049842947Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-17T09:09:24.079 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.052691586Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-17T09:09:24.079 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.055312441Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-17T09:09:24.079 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.057852283Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-17T09:09:24.079 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.058387282Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-17T09:09:24.079 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.061190078Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-17T09:09:24.079 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.06362978Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-17T09:09:24.079 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.066280963Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-17T09:09:24.079 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.069049945Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-17T09:09:24.079 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.071636686Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-17T09:09:24.322 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.074210008Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-17T09:09:24.322 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.0767723Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-17T09:09:24.322 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.077229231Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.079871142Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.080307725Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.082560222Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.082875149Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.083401676Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.083902259Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.084829678Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.085491787Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.086281383Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.086621431Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.087464161Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-17T09:09:24.323 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.088243015Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-17T09:09:24.324 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.089067581Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-17T09:09:24.324 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.089818958Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-17T09:09:24.324 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.090703556Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-17T09:09:24.324 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.091676464Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-17T09:09:24.325 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.092481326Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T09:09:24.325 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.092918325Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T09:09:24.325 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.093717963Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-17T09:09:24.325 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.094652659Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T09:09:24.325 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.095382806Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-17T09:09:24.325 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.095872554Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-17T09:09:24.325 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.096303331Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-17T09:09:24.325 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.096705101Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.097028597Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.097584144Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.098579552Z level=info msg="Executing migration" id="create data_source table" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.099225035Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.099992853Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.100841105Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.101688847Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.102248035Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.103278901Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.104104879Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.104883083Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-17T09:09:24.326 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.105291478Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.10619823Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.106900957Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.108852903Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.111600147Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.111936322Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.114844845Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.117476025Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.120071458Z level=info msg="Executing migration" id="Add uid column" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.120634594Z level=info msg="Executing migration" id="Update uid value" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.12354674Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.126008879Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.12884133Z level=info msg="Executing migration" id="create api_key table" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.131575869Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.134218939Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-17T09:09:24.327 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.136776167Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.139544499Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.13999301Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.14290151Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.145735458Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.148365465Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.148776215Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.149547048Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.150363903Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.151299355Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.151909083Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.152282384Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.153048893Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-17T09:09:24.328 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.153665648Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.154227225Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.15503849Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.155895074Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.156770488Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.15759279Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.158488276Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.159195912Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.159967026Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.16104922Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.161642482Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.162549489Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.163289706Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T09:09:24.329 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.164145606Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.165001443Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.165955106Z level=info msg="Executing migration" id="create quota table v1" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.166655904Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.167544408Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.170240821Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.170869602Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.173638265Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.17645792Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.178930253Z level=info msg="Executing migration" id="create session table" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.179399063Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.179700543Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.182102819Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-17T09:09:24.330 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.182558395Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.185354779Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.188171049Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.188569108Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.188866639Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.189192646Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.189602743Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.18987351Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.192356836Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.194921601Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.195571531Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.198516742Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.201062168Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-17T09:09:24.331 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.201555061Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.204464772Z level=info msg="Executing migration" id="create alert table v1" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.207040713Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.20984864Z level=info msg="Executing migration" id="add index alert state" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.212493254Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.213124934Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.213526824Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.213966964Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.214382175Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.215776575Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.216184236Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.216607273Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-17T09:09:24.332 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.216948874Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.217317922Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.217688463Z level=info msg="Executing migration" id="Add column is_default" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.21833615Z level=info msg="Executing migration" id="Add column frequency" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.21898213Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.219638282Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.220309264Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.22073752Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.221055121Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.22133163Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.22175727Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.222199043Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.222570661Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.222979334Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T09:09:24.333 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.223416566Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-17T09:09:24.334 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.224082476Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-17T09:09:24.334 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.224708183Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-17T09:09:24.334 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.225058886Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-17T09:09:24.334 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.225489999Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.225865298Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.226516914Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.226809704Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.227246202Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.227657523Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.228012083Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.228460685Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.228885318Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.229350091Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.22977787Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.231180846Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.233950491Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-17T09:09:24.335 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.236600456Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.237284027Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.239936974Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.242571627Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.245262633Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.247927862Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.250613674Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.253093328Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.255780558Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.258176892Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.260799129Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-17T09:09:24.336 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.263550421Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.266103706Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.268920131Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.271512853Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.274105667Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.276859431Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.279480932Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.281875231Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.282649176Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.283426361Z level=info msg="Executing migration" id="Move region to single row" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.284253942Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.28502384Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.285949855Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T09:09:24.337 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.286747448Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.287413407Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.288201781Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.288690746Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.289058538Z level=info msg="Executing migration" id="create test_data table" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.289829211Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.290263618Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.290746689Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.291480267Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.292301213Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.292822338Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.29313903Z level=info msg="Executing migration" id="create team table" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.29356785Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.294412006Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-17T09:09:24.338 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.29540221Z level=info msg="Executing migration" id="create team member table" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.296043406Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.296497077Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.297380248Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.298315508Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.299266127Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.299988165Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.300780838Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.301616538Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.302683731Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.303194953Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.306110633Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.308833581Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-17T09:09:24.339 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.311270294Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-17T09:09:24.340 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.314076328Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-17T09:09:24.340 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.316900572Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-17T09:09:24.340 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.319508233Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-17T09:09:24.340 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.32196998Z level=info msg="Executing migration" id="create tag table" 2024-09-17T09:09:24.340 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.324669286Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-17T09:09:24.576 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.327235759Z level=info msg="Executing migration" id="create login attempt table" 2024-09-17T09:09:24.576 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.32970202Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-17T09:09:24.576 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.332427028Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-17T09:09:24.576 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.33515609Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T09:09:24.577 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.337971096Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-17T09:09:24.577 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.340534645Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-17T09:09:24.577 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.343100613Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-17T09:09:24.577 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.345686526Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-17T09:09:24.577 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.348436108Z level=info msg="Executing migration" id="create user auth table" 2024-09-17T09:09:24.577 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.35106784Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T09:09:24.578 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.351929805Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-17T09:09:24.578 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.352560543Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-17T09:09:24.578 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.353473445Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-17T09:09:24.578 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.354430875Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-17T09:09:24.579 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.355209963Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-17T09:09:24.579 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.356005946Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-17T09:09:24.579 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.356953058Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-17T09:09:24.579 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.357892026Z level=info msg="Executing migration" id="create server_lock table" 2024-09-17T09:09:24.579 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.35870748Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-17T09:09:24.580 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.359636558Z level=info msg="Executing migration" id="create user auth token table" 2024-09-17T09:09:24.580 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.360285524Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-17T09:09:24.580 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.361099282Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-17T09:09:24.580 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.361899612Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-17T09:09:24.580 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.362680125Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-17T09:09:24.581 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.363492905Z level=info msg="Executing migration" id="create cache_data table" 2024-09-17T09:09:24.581 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.363908351Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-17T09:09:24.581 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.364778021Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-17T09:09:24.581 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.365599215Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-17T09:09:24.581 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.36640008Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-17T09:09:24.581 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.36673064Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-17T09:09:24.582 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.36756085Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-17T09:09:24.582 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.368440956Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-17T09:09:24.582 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.369279998Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-17T09:09:24.582 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.369777968Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-17T09:09:24.582 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.370733461Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-17T09:09:24.582 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.371623746Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-17T09:09:24.582 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.372257821Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-17T09:09:24.583 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.372689923Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-17T09:09:24.583 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.373144185Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-17T09:09:24.583 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.373991226Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-17T09:09:24.583 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.374937658Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-17T09:09:24.583 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.375652403Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-17T09:09:24.583 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.376088932Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-17T09:09:24.583 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.376986963Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-17T09:09:24.583 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.377873814Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-17T09:09:24.584 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.38064658Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-17T09:09:24.584 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.381175333Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-17T09:09:24.584 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.384037078Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-17T09:09:24.584 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.384560833Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-17T09:09:24.584 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.386984109Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-17T09:09:24.584 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.3894558Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-17T09:09:24.584 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.389899434Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-17T09:09:24.584 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.39269323Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-17T09:09:24.585 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.396178937Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-17T09:09:24.585 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.39951586Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-17T09:09:24.587 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.399994437Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-17T09:09:24.587 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.402860542Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-17T09:09:24.587 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.405685842Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-17T09:09:24.588 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.408617248Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-17T09:09:24.588 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.409056138Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-17T09:09:24.588 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.412057444Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-17T09:09:24.588 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.41485812Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-17T09:09:24.588 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.41729932Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-17T09:09:24.588 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.418206429Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.420922883Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.423714748Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.426194175Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.426731508Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.429365615Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.432084076Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.434464387Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.435336104Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.438009864Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.44083906Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-17T09:09:24.589 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.443301551Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.443764144Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.446395128Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.448912136Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.451641069Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.452520779Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.45335479Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.454184622Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.455044256Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.45535205Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.45617835Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.457019203Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.457863876Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.458722301Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-17T09:09:24.590 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.459648932Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.460713295Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.461574579Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.462037701Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.462886363Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.463632676Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.464561385Z level=info msg="Executing migration" id="create alert_image table" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.465148402Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.465979963Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.466966656Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.467641929Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.468056433Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T09:09:24.591 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.46813195Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.469016896Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.4698598Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.470784575Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.471057772Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.471168064Z level=info msg="alerts found to migrate" alerts=0 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.471556044Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.472548514Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.473171253Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.473968563Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.474409875Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.475316185Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-17T09:09:24.592 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.476048019Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.476932282Z level=info msg="Executing migration" id="create data_keys table" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.477715086Z level=info msg="Executing migration" id="create secrets table" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.478584768Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.483039839Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.483953951Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.484264264Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.488494583Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.492737525Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.495514573Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.495978993Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.496928145Z level=info msg="Executing migration" id="create permission table" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.497677809Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.498626572Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-17T09:09:24.593 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.499130518Z level=info msg="Executing migration" id="create role table" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.499914603Z level=info msg="Executing migration" id="add column display_name" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.50261631Z level=info msg="Executing migration" id="add column group_name" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.50529002Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.50575488Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.508485712Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.511065306Z level=info msg="Executing migration" id="create team role table" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.513820773Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.516648254Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.519295481Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.521960388Z level=info msg="Executing migration" id="create user role table" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.524626684Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.525080287Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.527835156Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.530575448Z level=info msg="Executing migration" id="create builtin role table" 2024-09-17T09:09:24.594 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.533137951Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.533580971Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.536396783Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.539159065Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.53982108Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.540246218Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.540618011Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.541045321Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.543743794Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.544284162Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.545293756Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.545710386Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.546166677Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.546468115Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-17T09:09:24.595 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.546849673Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.547305632Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.547649575Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.547967686Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.548310996Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.54868979Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.549114789Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.550064663Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.550360336Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.550770168Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.551198589Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.551608436Z level=info msg="Executing migration" id="add correlation config column" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.55258091Z level=info msg="Executing migration" id="create entity_events table" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.553024638Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-17T09:09:24.596 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.553447572Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.55352421Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.55380331Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.55388108Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.554142437Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.554522983Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.554929601Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.555354033Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.55577267Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.55618397Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.556567537Z level=info msg="Executing migration" id="Drop public config table" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.556987247Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.557404413Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-17T09:09:24.597 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.557807005Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T09:09:24.598 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.55826539Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-17T09:09:24.598 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.560973156Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-17T09:09:24.598 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.564015795Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-17T09:09:24.598 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.56501293Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-17T09:09:24.598 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.566027153Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-17T09:09:24.598 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.566342022Z level=info msg="Executing migration" id="add share column" 2024-09-17T09:09:24.599 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.567377309Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-17T09:09:24.599 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.56782923Z level=info msg="Executing migration" id="create file table" 2024-09-17T09:09:24.599 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.568273952Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-17T09:09:24.599 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.571070232Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-17T09:09:24.599 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.571596456Z level=info msg="Executing migration" id="create file_meta table" 2024-09-17T09:09:24.599 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.572039321Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-17T09:09:24.599 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.5747256Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-17T09:09:24.599 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.577178222Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-17T09:09:24.600 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.579609164Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-17T09:09:24.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.580014561Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.582700945Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.585274655Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.587918738Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.588365228Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.590972847Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.591307995Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.593753131Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.594865586Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.596164743Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.596676068Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.606647435Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-17T09:09:24.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.607385947Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.608284378Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.611982975Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.61288917Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.613580651Z level=info msg="Executing migration" id="create folder table" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.614508676Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.615060905Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.615876173Z level=info msg="Executing migration" id="Update folder title length" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.616805209Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=migrator t=2024-09-17T09:09:24.617388105Z level=info msg="migrations completed" performed=485 skipped=0 duration=636.639032ms 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=sqlstore t=2024-09-17T09:09:24.617834178Z level=info msg="Created default organization" 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=plugin.loader t=2024-09-17T09:09:24.672609201Z level=info msg="Plugin registered" pluginID=input 2024-09-17T09:09:24.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=plugin.loader t=2024-09-17T09:09:24.701101037Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-17T09:09:24.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=plugin.loader t=2024-09-17T09:09:24.701133105Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-17T09:09:24.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=secrets t=2024-09-17T09:09:24.701165499Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-17T09:09:24.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=query_data t=2024-09-17T09:09:24.702855243Z level=info msg="Query Service initialization" 2024-09-17T09:09:24.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=live.push_http t=2024-09-17T09:09:24.705210605Z level=info msg="Live Push Gateway initialization" 2024-09-17T09:09:24.971 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:24 smithi079 bash[78105]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T09:09:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[40839]: Deploying daemon node-exporter.a on smithi079 2024-09-17T09:09:25.238 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.238 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.238 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.238 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.238 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:24 smithi079 ceph-mon[29577]: Deploying daemon node-exporter.a on smithi079 2024-09-17T09:09:25.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:25.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:24 smithi173 ceph-mon[33421]: Deploying daemon node-exporter.a on smithi079 2024-09-17T09:09:25.611 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:25 smithi079 bash[78105]: Getting image source signatures 2024-09-17T09:09:25.611 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:25 smithi079 bash[78105]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T09:09:25.611 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:25 smithi079 bash[78105]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T09:09:25.612 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:25 smithi079 bash[78105]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T09:09:25.756 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=infra.usagestats.collector t=2024-09-17T09:09:25.413800174Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=provisioning.datasources t=2024-09-17T09:09:25.46800973Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=provisioning.datasources t=2024-09-17T09:09:25.499881098Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=provisioning.alerting t=2024-09-17T09:09:25.521221381Z level=info msg="starting to provision alerting" 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=provisioning.alerting t=2024-09-17T09:09:25.521258482Z level=info msg="finished to provision alerting" 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=grafanaStorageLogger t=2024-09-17T09:09:25.521439671Z level=info msg="storage starting" 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=ngalert.state.manager t=2024-09-17T09:09:25.521440085Z level=info msg="Warming state cache for startup" 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=http.server t=2024-09-17T09:09:25.523391517Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=sqlstore.transactions t=2024-09-17T09:09:25.532485802Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=ngalert.state.manager t=2024-09-17T09:09:25.53255369Z level=info msg="State cache has been initialized" states=0 duration=11.113117ms 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=ticker t=2024-09-17T09:09:25.532613403Z level=info msg=starting first_tick=2024-09-17T09:09:30Z 2024-09-17T09:09:25.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=ngalert.multiorg.alertmanager t=2024-09-17T09:09:25.532611454Z level=info msg="starting MultiOrg Alertmanager" 2024-09-17T09:09:25.758 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=sqlstore.transactions t=2024-09-17T09:09:25.575849089Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T09:09:25.758 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=sqlstore.transactions t=2024-09-17T09:09:25.58706079Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-17T09:09:25.758 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a[66527]: logger=sqlstore.transactions t=2024-09-17T09:09:25.622972677Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T09:09:26.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:25 smithi173 ceph-mon[33421]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:26.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:25 smithi079 ceph-mon[29577]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:26.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:25 smithi079 ceph-mon[40839]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:26.861 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:26 smithi079 bash[78105]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T09:09:26.861 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:26 smithi079 bash[78105]: Writing manifest to image destination 2024-09-17T09:09:27.247 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:26 smithi079 podman[78105]: 2024-09-17 09:09:26.904741947 +0000 UTC m=+2.110167945 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T09:09:27.247 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 podman[78105]: 2024-09-17 09:09:27.011445227 +0000 UTC m=+2.216871213 container create c51a61eee4484a9d5b26f34153102b912831142958269b3ca6ca52ea0d48bc92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:27.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.613 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 podman[78105]: 2024-09-17 09:09:27.246872245 +0000 UTC m=+2.452298234 container init c51a61eee4484a9d5b26f34153102b912831142958269b3ca6ca52ea0d48bc92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:27.613 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 podman[78105]: 2024-09-17 09:09:27.252376787 +0000 UTC m=+2.457802773 container start c51a61eee4484a9d5b26f34153102b912831142958269b3ca6ca52ea0d48bc92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:27.613 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.256Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T09:09:27.613 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.256Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T09:09:27.613 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.257Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T09:09:27.613 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.257Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T09:09:27.613 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.257Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T09:09:27.613 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.257Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T09:09:27.614 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T09:09:27.615 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T09:09:27.616 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a[78182]: ts=2024-09-17T09:09:27.258Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 bash[78105]: c51a61eee4484a9d5b26f34153102b912831142958269b3ca6ca52ea0d48bc92 2024-09-17T09:09:27.617 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:09:27 smithi079 systemd[1]: Started Ceph node-exporter.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:09:27.753 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:27 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.753 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:27 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.753 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:27 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:27.753 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:27 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:28.373 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:09:28.416 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:28 smithi173 bash[66795]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T09:09:28.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:28 smithi079 ceph-mon[29577]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:28.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:28 smithi079 ceph-mon[29577]: Deploying daemon node-exporter.b on smithi173 2024-09-17T09:09:28.699 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:28 smithi079 ceph-mon[40839]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:28.699 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:28 smithi079 ceph-mon[40839]: Deploying daemon node-exporter.b on smithi173 2024-09-17T09:09:28.699 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:09:28.699 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:09:28.699 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:09:28.722 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:28 smithi173 ceph-mon[33421]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:28.723 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:28 smithi173 ceph-mon[33421]: Deploying daemon node-exporter.b on smithi173 2024-09-17T09:09:29.006 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:28 smithi173 bash[66795]: Getting image source signatures 2024-09-17T09:09:29.006 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:28 smithi173 bash[66795]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T09:09:29.006 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:28 smithi173 bash[66795]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T09:09:29.006 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:28 smithi173 bash[66795]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T09:09:29.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:29 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3540575708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:29.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:29 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3540575708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:29.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:29 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3540575708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:30.422 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 bash[66795]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T09:09:30.422 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 bash[66795]: Writing manifest to image destination 2024-09-17T09:09:30.740 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-mon[33421]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:30.741 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 podman[66795]: 2024-09-17 09:09:30.525558901 +0000 UTC m=+2.282777563 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T09:09:30.741 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 podman[66795]: 2024-09-17 09:09:30.632257345 +0000 UTC m=+2.389476001 container create eb913e3b3a96f8cb6aeeaef1ccf8a45f048c64564a7b22847ce21616cd1ed59e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:09:30.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:30 smithi079 ceph-mon[29577]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:30.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:30 smithi079 ceph-mon[40839]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 podman[66795]: 2024-09-17 09:09:30.842784798 +0000 UTC m=+2.600003447 container init eb913e3b3a96f8cb6aeeaef1ccf8a45f048c64564a7b22847ce21616cd1ed59e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 podman[66795]: 2024-09-17 09:09:30.847919248 +0000 UTC m=+2.605137896 container start eb913e3b3a96f8cb6aeeaef1ccf8a45f048c64564a7b22847ce21616cd1ed59e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.851Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.851Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.852Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.852Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.852Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.852Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T09:09:31.257 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T09:09:31.258 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T09:09:31.258 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T09:09:31.258 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T09:09:31.258 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T09:09:31.258 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T09:09:31.258 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T09:09:31.258 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T09:09:31.259 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T09:09:31.260 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.853Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.854Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b[66870]: ts=2024-09-17T09:09:30.854Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 bash[66795]: eb913e3b3a96f8cb6aeeaef1ccf8a45f048c64564a7b22847ce21616cd1ed59e 2024-09-17T09:09:31.261 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:09:30 smithi173 systemd[1]: Started Ceph node-exporter.b for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-17T09:09:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:32.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:32.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:32.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:32.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:09:32 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:09:33.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:33 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-17T09:09:33.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:33 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-17T09:09:33.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:33 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-17T09:09:33.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:33 smithi173 ceph-mon[33421]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T09:09:33.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:33 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:33.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-17T09:09:33.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-17T09:09:33.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-17T09:09:33.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[29577]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T09:09:33.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:33.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-17T09:09:33.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-17T09:09:33.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-17T09:09:33.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[40839]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T09:09:33.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:33.700 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:09:33.751 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:09:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:09:34.021 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:09:34.021 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:09:34.021 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:09:34.271 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[29577]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T09:09:34.271 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[29577]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:34.271 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:34.271 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:34.272 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3842022073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:34.272 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[29577]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-17T09:09:34.272 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[40839]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T09:09:34.272 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[40839]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:34.272 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:34.272 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:34.272 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3842022073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:34.272 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:34 smithi079 ceph-mon[40839]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-17T09:09:34.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:34 smithi173 ceph-mon[33421]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T09:09:34.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:34 smithi173 ceph-mon[33421]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:34.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:34 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:34.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:34 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:34.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:34 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3842022073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:34.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:34 smithi173 ceph-mon[33421]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-17T09:09:36.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:36 smithi173 ceph-mon[33421]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T09:09:36.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:36 smithi079 ceph-mon[29577]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T09:09:36.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:36 smithi079 ceph-mon[40839]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T09:09:37.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:37 smithi173 ceph-mon[33421]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T09:09:37.780 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:37 smithi173 ceph-mon[33421]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-17T09:09:37.780 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:37 smithi173 ceph-mon[33421]: Cluster is now healthy 2024-09-17T09:09:37.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:37 smithi079 ceph-mon[29577]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T09:09:37.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:37 smithi079 ceph-mon[29577]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-17T09:09:37.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:37 smithi079 ceph-mon[29577]: Cluster is now healthy 2024-09-17T09:09:37.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:37 smithi079 ceph-mon[40839]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T09:09:37.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:37 smithi079 ceph-mon[40839]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-17T09:09:37.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:37 smithi079 ceph-mon[40839]: Cluster is now healthy 2024-09-17T09:09:39.024 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:09:39.350 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:09:39.351 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:09:39.351 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:09:39.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:39 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/244273677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:39.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:39 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/244273677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:39.706 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:39 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/244273677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:40.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:40 smithi173 ceph-mon[33421]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T09:09:40.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:40 smithi079 ceph-mon[29577]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T09:09:40.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:40 smithi079 ceph-mon[40839]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: Rotating authentication key for osd.0 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]': finished 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: Reconfiguring daemon osd.0 on smithi079 2024-09-17T09:09:42.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[29577]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11 B/s, 1 objects/s recovering 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: Rotating authentication key for osd.0 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]': finished 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: Reconfiguring daemon osd.0 on smithi079 2024-09-17T09:09:42.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:41 smithi079 ceph-mon[40839]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11 B/s, 1 objects/s recovering 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: Rotating authentication key for osd.0 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]': finished 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: Reconfiguring daemon osd.0 on smithi079 2024-09-17T09:09:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:41 smithi173 ceph-mon[33421]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11 B/s, 1 objects/s recovering 2024-09-17T09:09:42.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:09:42 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:09:43.394 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:43.394 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.394 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.394 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.394 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.395 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:43.395 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.395 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.395 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.395 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:43 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:43.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:43 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:43 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:43 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:43 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:43.655 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:43 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:09:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:09:44.352 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:44 smithi079 ceph-mon[29577]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:09:44.353 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:44 smithi079 ceph-mon[29577]: Reconfiguring daemon alertmanager.a on smithi079 2024-09-17T09:09:44.353 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:44 smithi079 ceph-mon[29577]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-09-17T09:09:44.353 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:44 smithi079 ceph-mon[40839]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:09:44.353 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:44 smithi079 ceph-mon[40839]: Reconfiguring daemon alertmanager.a on smithi079 2024-09-17T09:09:44.353 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:44 smithi079 ceph-mon[40839]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-09-17T09:09:44.353 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:09:44.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:44 smithi173 ceph-mon[33421]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:09:44.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:44 smithi173 ceph-mon[33421]: Reconfiguring daemon alertmanager.a on smithi079 2024-09-17T09:09:44.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:44 smithi173 ceph-mon[33421]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-09-17T09:09:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:09:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:09:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:09:45.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:45 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1993125916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:45.517 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:45 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1993125916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:45.517 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:45 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1993125916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:46.111 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:45 smithi079 systemd[1]: Stopping Ceph alertmanager.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:09:46.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:46 smithi173 ceph-mon[33421]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9 B/s, 1 objects/s recovering 2024-09-17T09:09:46.529 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:46 smithi079 ceph-mon[29577]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9 B/s, 1 objects/s recovering 2024-09-17T09:09:46.530 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:46 smithi079 ceph-mon[40839]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9 B/s, 1 objects/s recovering 2024-09-17T09:09:46.530 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:46 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[76991]: ts=2024-09-17T09:09:46.206Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:09:46.530 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:46 smithi079 podman[79304]: 2024-09-17 09:09:46.227575235 +0000 UTC m=+0.120087104 container died 7d5c3b1d7b0eb38f2ad38c60432596182ad1515e7149dc57540fedb8628257b3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:46.530 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:46 smithi079 podman[79304]: 2024-09-17 09:09:46.530047796 +0000 UTC m=+0.422559671 container cleanup 7d5c3b1d7b0eb38f2ad38c60432596182ad1515e7149dc57540fedb8628257b3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:46.840 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:46 smithi079 bash[79304]: ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T09:09:46.840 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:46 smithi079 podman[79321]: 2024-09-17 09:09:46.739614533 +0000 UTC m=+0.509761531 container remove 7d5c3b1d7b0eb38f2ad38c60432596182ad1515e7149dc57540fedb8628257b3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:46.840 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:46 smithi079 podman[79321]: 2024-09-17 09:09:46.840107049 +0000 UTC m=+0.610254048 volume remove 869e6ddb23a5adcc0ba379dd47bb998649ad694c2fe4d8fb5cd6ce39244d7910 2024-09-17T09:09:47.111 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:46 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T09:09:47.111 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:46 smithi079 systemd[1]: Stopped Ceph alertmanager.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:09:47.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:46 smithi079 systemd[1]: Starting Ceph alertmanager.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:09:47.480 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:47.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:47.481 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 podman[79417]: 2024-09-17 09:09:47.083240471 +0000 UTC m=+0.015722693 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-17T09:09:47.481 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 podman[79417]: 2024-09-17 09:09:47.199806887 +0000 UTC m=+0.132289097 volume create c6c1339e4580278ab77261fbdc8250a522c5d879b96eea658da59bbac89f8644 2024-09-17T09:09:47.481 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 podman[79417]: 2024-09-17 09:09:47.291812982 +0000 UTC m=+0.224295191 container create 21e4ae85c61a83a9799fe953541f9483885fc25f1a4393112359e05cff9c06a3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:47.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:47 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:09:47.862 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 podman[79417]: 2024-09-17 09:09:47.578144548 +0000 UTC m=+0.510626757 container init 21e4ae85c61a83a9799fe953541f9483885fc25f1a4393112359e05cff9c06a3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:47.862 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 podman[79417]: 2024-09-17 09:09:47.583173106 +0000 UTC m=+0.515655317 container start 21e4ae85c61a83a9799fe953541f9483885fc25f1a4393112359e05cff9c06a3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:47.862 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:47.594Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-17T09:09:47.862 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:47.594Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-17T09:09:47.862 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:47.596Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.79 port=9094 2024-09-17T09:09:47.862 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:47.597Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T09:09:47.862 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:47.619Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:09:47.863 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:47.620Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:09:47.863 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:47.622Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-17T09:09:47.863 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:47.622Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-17T09:09:47.863 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 bash[79417]: 21e4ae85c61a83a9799fe953541f9483885fc25f1a4393112359e05cff9c06a3 2024-09-17T09:09:47.863 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:47 smithi079 systemd[1]: Started Ceph alertmanager.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:09:48.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:48 smithi173 ceph-mon[33421]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 9 B/s, 1 objects/s recovering 2024-09-17T09:09:48.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:48 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:48.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:48 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:48.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:48 smithi079 ceph-mon[40839]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 9 B/s, 1 objects/s recovering 2024-09-17T09:09:48.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:48 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:48.613 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:48 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:48.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:48 smithi079 ceph-mon[29577]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 9 B/s, 1 objects/s recovering 2024-09-17T09:09:48.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:48 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:48.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:48 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:49.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:49 smithi173 ceph-mon[33421]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:09:49.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:49 smithi173 ceph-mon[33421]: Reconfiguring daemon prometheus.a on smithi173 2024-09-17T09:09:49.597 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:49 smithi079 ceph-mon[29577]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:09:49.597 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:49 smithi079 ceph-mon[29577]: Reconfiguring daemon prometheus.a on smithi173 2024-09-17T09:09:49.598 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:49 smithi079 ceph-mon[40839]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:09:49.598 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:49 smithi079 ceph-mon[40839]: Reconfiguring daemon prometheus.a on smithi173 2024-09-17T09:09:49.679 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:09:49.861 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:49 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:49.597Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000330952s 2024-09-17T09:09:50.004 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:09:50.004 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:09:50.004 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:09:50.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:50 smithi079 ceph-mon[29577]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:50.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:50 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1201916054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:50.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:50 smithi079 ceph-mon[40839]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:50.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:50 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1201916054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:50.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:50 smithi173 ceph-mon[33421]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:50.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:50 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1201916054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:51.256 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 systemd[1]: Stopping Ceph prometheus.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:09:51.624 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.341Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:09:51.624 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.341Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.341Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.341Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.341Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.341Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.341Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.341Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.342Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.342Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.342Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[64543]: ts=2024-09-17T09:09:51.342Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T09:09:51.625 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 podman[67907]: 2024-09-17 09:09:51.351984432 +0000 UTC m=+0.126650572 container died 5efcab213104dc40ed8bcbfe2587e0fe5791798844176acad6c2673ea2af6da0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:52.006 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 podman[67907]: 2024-09-17 09:09:51.73065833 +0000 UTC m=+0.505324466 container cleanup 5efcab213104dc40ed8bcbfe2587e0fe5791798844176acad6c2673ea2af6da0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:52.006 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:51 smithi173 bash[67907]: ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T09:09:52.361 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 podman[67925]: 2024-09-17 09:09:52.015067975 +0000 UTC m=+0.664117795 container remove 5efcab213104dc40ed8bcbfe2587e0fe5791798844176acad6c2673ea2af6da0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:52.361 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T09:09:52.361 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 systemd[1]: Stopped Ceph prometheus.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:09:52.361 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.167s CPU time. 2024-09-17T09:09:52.361 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 systemd[1]: Starting Ceph prometheus.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:09:52.643 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-mon[33421]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:52.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:09:52.644 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 podman[68024]: 2024-09-17 09:09:52.264228235 +0000 UTC m=+0.016791542 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-17T09:09:52.644 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 podman[68024]: 2024-09-17 09:09:52.407639799 +0000 UTC m=+0.160203140 container create 6e0ce71149469d8f8baa9ef53c5640c1a7648a71fb59d3bcb7c8337f78e26e8b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:52.644 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 podman[68024]: 2024-09-17 09:09:52.643149526 +0000 UTC m=+0.395712832 container init 6e0ce71149469d8f8baa9ef53c5640c1a7648a71fb59d3bcb7c8337f78e26e8b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:52.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:52 smithi079 ceph-mon[29577]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:52.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:52 smithi079 ceph-mon[40839]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:53.007 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 podman[68024]: 2024-09-17 09:09:52.648140061 +0000 UTC m=+0.400703351 container start 6e0ce71149469d8f8baa9ef53c5640c1a7648a71fb59d3bcb7c8337f78e26e8b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:09:53.007 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.672Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T09:09:53.007 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.672Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T09:09:53.007 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.672Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi173 (none))" 2024-09-17T09:09:53.007 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.672Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:09:53.007 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.672Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:09:53.007 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.674Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.674Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.675Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.675Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.677Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.677Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.686µs 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.677Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.677Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.677Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.677Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.151µs wal_replay_duration=578.697µs wbl_replay_duration=100ns total_replay_duration=618.154µs 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.678Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.678Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T09:09:53.008 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.678Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:09:53.009 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.698Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.041074ms db_storage=724ns remote_storage=909ns web_handler=389ns query_engine=545ns scrape=395.791µs scrape_sd=78.901µs notify=13.989µs notify_sd=33.919µs rules=19.171002ms tracing=6.258µs 2024-09-17T09:09:53.009 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.698Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T09:09:53.009 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:09:52.698Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T09:09:53.009 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 bash[68024]: 6e0ce71149469d8f8baa9ef53c5640c1a7648a71fb59d3bcb7c8337f78e26e8b 2024-09-17T09:09:53.009 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:09:52 smithi173 systemd[1]: Started Ceph prometheus.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:09:53.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:52 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:52] ENGINE Bus STOPPING 2024-09-17T09:09:53.123 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:52 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T09:09:53.123 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:52 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:52] ENGINE Bus STOPPED 2024-09-17T09:09:53.123 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:52 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:52] ENGINE Bus STARTING 2024-09-17T09:09:53.123 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:52 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:52] ENGINE Serving on http://:::9283 2024-09-17T09:09:53.123 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:52 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:52] ENGINE Bus STARTED 2024-09-17T09:09:53.124 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:52 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:52] ENGINE Bus STOPPING 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:09:53.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:09:53.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:53 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:09:53.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:53.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:53] ENGINE Bus STOPPED 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:53] ENGINE Bus STARTING 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:53] ENGINE Serving on http://:::9283 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:53] ENGINE Bus STARTED 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:53] ENGINE Bus STOPPING 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:09:53.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:09:53.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:09:53.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:09:53.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:09:53.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:09:53.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:53.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:53 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:09:54.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T09:09:54.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:54] ENGINE Bus STOPPED 2024-09-17T09:09:54.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:54] ENGINE Bus STARTING 2024-09-17T09:09:54.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:54] ENGINE Serving on http://:::9283 2024-09-17T09:09:54.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:09:54] ENGINE Bus STARTED 2024-09-17T09:09:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:54 smithi173 ceph-mon[33421]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:09:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:54 smithi173 ceph-mon[33421]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:09:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:54 smithi173 ceph-mon[33421]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:09:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:54 smithi173 ceph-mon[33421]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:09:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:54 smithi173 ceph-mon[33421]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:09:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:54 smithi173 ceph-mon[33421]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:09:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:54 smithi173 ceph-mon[33421]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:54.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[40839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:09:54.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[40839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:09:54.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[40839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:09:54.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[40839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[40839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[40839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[40839]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:09:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:54 smithi079 ceph-mon[29577]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:55.006 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:09:55.333 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:09:55.333 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:09:55.334 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:09:55.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:55 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2556520898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:55.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:55 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2556520898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:55.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:55 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2556520898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:09:56.726 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:56 smithi173 ceph-mon[33421]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:56.726 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:56 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:56.726 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:56 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:56.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:56 smithi079 ceph-mon[40839]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:56.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:56 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:56.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:56 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:56.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:56 smithi079 ceph-mon[29577]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:09:56.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:56 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:56.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:56 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:57.861 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:09:57.599Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001843987s 2024-09-17T09:09:58.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:57 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:58.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:57 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:58.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:57 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:09:58.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:57 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:09:58.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:57 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:58.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:09:57 smithi173 ceph-mon[33421]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:58.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:58.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[29577]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:09:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:09:57 smithi079 ceph-mon[40839]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:00.335 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:00.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:00 smithi079 ceph-mon[29577]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:00.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:00 smithi079 ceph-mon[29577]: overall HEALTH_OK 2024-09-17T09:10:00.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:00 smithi079 ceph-mon[40839]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:00.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:00 smithi079 ceph-mon[40839]: overall HEALTH_OK 2024-09-17T09:10:00.661 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:00.661 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:00.661 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:00.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:00 smithi173 ceph-mon[33421]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:00.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:00 smithi173 ceph-mon[33421]: overall HEALTH_OK 2024-09-17T09:10:01.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:01 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1024327114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:01.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:01 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1024327114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:01.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:01 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1024327114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:02.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:02 smithi173 ceph-mon[33421]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:02.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:02.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:10:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:10:02.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:02 smithi079 ceph-mon[29577]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:02.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:02 smithi079 ceph-mon[40839]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:03.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:03 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:03.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:03 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:03.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:10:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:10:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:10:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:03 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:04.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:04 smithi173 ceph-mon[33421]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:04.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:04 smithi079 ceph-mon[29577]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:04.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:04 smithi079 ceph-mon[40839]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:05.662 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:05.996 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:05.996 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:05.996 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:06.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:06 smithi173 ceph-mon[33421]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:06.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:06 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2976007044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:06.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:06 smithi079 ceph-mon[40839]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:06.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:06 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2976007044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:06.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:06 smithi079 ceph-mon[29577]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:06.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:06 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2976007044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:08.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:08 smithi173 ceph-mon[33421]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:08 smithi079 ceph-mon[29577]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:08.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:08 smithi079 ceph-mon[40839]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:10.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:10 smithi173 ceph-mon[33421]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:10.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:10 smithi079 ceph-mon[29577]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:10.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:10 smithi079 ceph-mon[40839]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:10.999 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:11.324 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:11.325 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:11.325 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:11.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:11 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2350239878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:11.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:11 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2350239878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:11.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:11 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2350239878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:12.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:12 smithi173 ceph-mon[33421]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:12.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:12 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:12.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:10:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:10:12.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:12 smithi079 ceph-mon[29577]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:12.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:12 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:12.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:12 smithi079 ceph-mon[40839]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:12.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:12 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:13.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:13 smithi173 ceph-mon[33421]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:13.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:13 smithi079 ceph-mon[29577]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:13.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:13 smithi079 ceph-mon[40839]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:13.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:10:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:10:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:10:16.326 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:16.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:16 smithi079 ceph-mon[29577]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:16.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:16 smithi079 ceph-mon[40839]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:16.654 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:16.654 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:16.654 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:16.722 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:16 smithi173 ceph-mon[33421]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:17.693 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:17 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:17.693 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:17 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/4085005017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:17.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:17 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:17.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:17 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4085005017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:17.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:17 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:17.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:17 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/4085005017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:18.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:18 smithi173 ceph-mon[33421]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:18.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:18 smithi079 ceph-mon[29577]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:18.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:18 smithi079 ceph-mon[40839]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:20.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:20 smithi173 ceph-mon[33421]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:20.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:20 smithi079 ceph-mon[29577]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:20.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:20 smithi079 ceph-mon[40839]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:21.656 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:21.983 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:21.983 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:21.983 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:22.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:22 smithi173 ceph-mon[33421]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:22.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:22 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2229583012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:22.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:10:22 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:10:22.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:22 smithi079 ceph-mon[29577]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:22.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:22 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2229583012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:22.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:22 smithi079 ceph-mon[40839]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:22.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:22 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2229583012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:23.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:10:23 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:10:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:10:24.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:24 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:24.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:24 smithi173 ceph-mon[33421]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:24 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:24 smithi079 ceph-mon[29577]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:24 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:24 smithi079 ceph-mon[40839]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:26.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:26 smithi173 ceph-mon[33421]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:26.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:26 smithi079 ceph-mon[29577]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:26.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:26 smithi079 ceph-mon[40839]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:26.985 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:27.308 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:27.309 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:27.309 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:27.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:27 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3216569109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:27.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:27 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3216569109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:27.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:27 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3216569109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:28.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:28 smithi173 ceph-mon[33421]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:28.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:28 smithi079 ceph-mon[29577]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:28.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:28 smithi079 ceph-mon[40839]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:29.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:29 smithi173 ceph-mon[33421]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:29.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:29 smithi079 ceph-mon[29577]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:29.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:29 smithi079 ceph-mon[40839]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:32.310 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:32.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:32 smithi079 ceph-mon[29577]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:32.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:32.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:32 smithi079 ceph-mon[40839]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:32.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:32.644 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:32.644 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:32.644 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:32.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:32 smithi173 ceph-mon[33421]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:32.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:32.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:10:32 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:10:33.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:33 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3797389771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:33.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:33 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3797389771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:33.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:33 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3797389771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:33.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:10:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:10:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:10:34.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:34 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:34 smithi173 ceph-mon[33421]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:34.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:34 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:34.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:34 smithi079 ceph-mon[29577]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:34.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:34 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:34.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:34 smithi079 ceph-mon[40839]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:36.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:36 smithi079 ceph-mon[29577]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:36.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:36 smithi079 ceph-mon[40839]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:36.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:36 smithi173 ceph-mon[33421]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:37.646 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:37.962 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:37.962 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:37.962 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:38.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:38 smithi173 ceph-mon[33421]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:38.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:38 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2317562258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:38.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:38 smithi079 ceph-mon[29577]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:38.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:38 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2317562258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:38.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:38 smithi079 ceph-mon[40839]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:38.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:38 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2317562258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:40.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:40 smithi173 ceph-mon[33421]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:40.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:40 smithi079 ceph-mon[29577]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:40.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:40 smithi079 ceph-mon[40839]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:42.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:42 smithi173 ceph-mon[33421]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:42.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:10:42 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:10:42.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:42 smithi079 ceph-mon[29577]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:42.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:42 smithi079 ceph-mon[40839]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:42.964 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:43.291 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:43.291 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:43.292 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:43.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:43 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/302760118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:43.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:43 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/302760118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:43.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:10:43 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:10:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:10:43.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:43 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/302760118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:44.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:44 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:44.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:44 smithi173 ceph-mon[33421]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:44.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:44 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:44.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:44 smithi079 ceph-mon[29577]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:44.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:44 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:44.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:44 smithi079 ceph-mon[40839]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:45.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:45 smithi079 ceph-mon[29577]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:45.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:45 smithi079 ceph-mon[40839]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:46.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:45 smithi173 ceph-mon[33421]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:46.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:46 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:46.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:46 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:46.907 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:46 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:10:47.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:47 smithi079 ceph-mon[29577]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:47.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:47 smithi079 ceph-mon[40839]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:48.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:47 smithi173 ceph-mon[33421]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:48.293 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:48.621 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:48.621 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:48.621 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:50.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:49 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/390721481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:50.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:49 smithi173 ceph-mon[33421]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:50.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:49 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/390721481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:50.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:49 smithi079 ceph-mon[29577]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:49 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/390721481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:50.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:49 smithi079 ceph-mon[40839]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:52.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:52 smithi173 ceph-mon[33421]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:52.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:10:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:10:52.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:52 smithi079 ceph-mon[29577]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:52.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:52 smithi079 ceph-mon[40839]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:53.623 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:53.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:10:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:10:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:10:53.949 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:53.949 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:53.949 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:54.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:54 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:54.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:54 smithi173 ceph-mon[33421]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:54 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3019769022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:54.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:54 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:54.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:54 smithi079 ceph-mon[40839]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:54.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:54 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3019769022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:54 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:10:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:54 smithi079 ceph-mon[29577]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:54 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3019769022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:56.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:56 smithi173 ceph-mon[33421]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:56.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:56 smithi079 ceph-mon[29577]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:56.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:56 smithi079 ceph-mon[40839]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:10:57.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:57 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:10:57.786 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:57 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:10:57.786 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:57 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:10:58.748 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:58 smithi079 ceph-mon[29577]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:58.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:58 smithi079 ceph-mon[40839]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:58.754 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:58 smithi173 ceph-mon[33421]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:10:58.948 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:10:59.285 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:10:59.285 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:10:59.285 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:10:59.544 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:10:59 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3264700964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:59.544 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:10:59 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3264700964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:10:59.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:10:59 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3264700964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:00 smithi173 ceph-mon[33421]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:00.740 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:00 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:00.740 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:00 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:00.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:00 smithi079 ceph-mon[29577]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:00.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:00 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:00.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:00 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:00.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:00 smithi079 ceph-mon[40839]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:00.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:00 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:00.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:00 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:01 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:01 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:01 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:11:02.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:01 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:11:02.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:01 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:01 smithi173 ceph-mon[33421]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:02.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:01 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:02.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:11:02.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:11:02.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[29577]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:02.366 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:02.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:11:02.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:11:02.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:11:02.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[40839]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:02.366 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:01 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:02.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:11:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:11:03.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:11:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:11:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:11:04.287 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:04.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:04 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:04.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:04 smithi079 ceph-mon[29577]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:04.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:04 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:04.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:04 smithi079 ceph-mon[40839]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:04.621 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:04.621 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:04.621 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:04.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:04 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:04.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:04 smithi173 ceph-mon[33421]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:05.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:05 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1988097436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:05.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:05 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1988097436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:05.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:05 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1988097436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:06.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:06 smithi173 ceph-mon[33421]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:06.814 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:06 smithi079 ceph-mon[29577]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:06.814 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:06 smithi079 ceph-mon[40839]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:08.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:08 smithi173 ceph-mon[33421]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:08.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:08 smithi079 ceph-mon[29577]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:08.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:08 smithi079 ceph-mon[40839]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:09.624 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:09.945 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:09.945 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:09.945 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:10.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:10 smithi173 ceph-mon[33421]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:10.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:10 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3970931055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:10.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:10 smithi079 ceph-mon[40839]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:10.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:10 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3970931055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:10.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:10 smithi079 ceph-mon[29577]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:10.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:10 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3970931055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:12.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:12 smithi173 ceph-mon[33421]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:12.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:11:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:11:12.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:12 smithi079 ceph-mon[29577]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:12.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:12 smithi079 ceph-mon[40839]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:13.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:11:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:11:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:11:14.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:14 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:14.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:14 smithi173 ceph-mon[33421]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:14.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:14 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:14.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:14 smithi079 ceph-mon[29577]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:14.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:14 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:14.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:14 smithi079 ceph-mon[40839]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:14.947 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:15.273 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:15.273 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:15.274 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:15.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:15 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/474972819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:15.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:15 smithi079 ceph-mon[29577]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:15.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:15 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/474972819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:15.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:15 smithi079 ceph-mon[40839]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:15.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:15 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/474972819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:15.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:15 smithi173 ceph-mon[33421]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:16.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:16 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:16.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:16 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:17.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:16 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:17.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:17 smithi079 ceph-mon[29577]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:17.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:17 smithi079 ceph-mon[40839]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:17.925 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:17 smithi173 ceph-mon[33421]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:20.275 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:20.606 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:20 smithi079 ceph-mon[29577]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:20.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:20 smithi079 ceph-mon[40839]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:20.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:20 smithi173 ceph-mon[33421]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:21.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:21 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/970203680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:21.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:21 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/970203680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:21.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:21 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/970203680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:22.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:22 smithi173 ceph-mon[33421]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:22.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:11:22 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:11:22.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:22 smithi079 ceph-mon[29577]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:22.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:22 smithi079 ceph-mon[40839]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:23.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:11:23 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:11:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:11:24.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:24 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:24.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:24 smithi173 ceph-mon[33421]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:24 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:24 smithi079 ceph-mon[29577]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:24 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:24 smithi079 ceph-mon[40839]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:25.608 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:25.932 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:25.932 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:25.932 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:26.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:26 smithi173 ceph-mon[33421]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:26.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:26 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/4145085565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:26.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:26 smithi079 ceph-mon[29577]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:26.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:26 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4145085565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:26.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:26 smithi079 ceph-mon[40839]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:26.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:26 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/4145085565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:28.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:28 smithi173 ceph-mon[33421]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:28.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:28 smithi079 ceph-mon[29577]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:28.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:28 smithi079 ceph-mon[40839]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:29.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:29 smithi173 ceph-mon[33421]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:29.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:29 smithi079 ceph-mon[29577]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:29.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:29 smithi079 ceph-mon[40839]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:30.933 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:31.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:31 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4180060200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:31.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:31 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/4180060200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:31.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:31 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/4180060200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:32.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:32 smithi079 ceph-mon[29577]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:32.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:32.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:32 smithi079 ceph-mon[40839]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:32.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:32.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:32 smithi173 ceph-mon[33421]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:32.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:32.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:11:32 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:11:33.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:11:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:11:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:11:34.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:34 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:34.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:34 smithi173 ceph-mon[33421]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:34.762 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:34 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:34.762 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:34 smithi079 ceph-mon[29577]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:34.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:34 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:34.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:34 smithi079 ceph-mon[40839]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:36.259 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:36.591 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:36.591 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:36.591 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:36.592 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:36 smithi079 ceph-mon[29577]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:36 smithi079 ceph-mon[40839]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:36.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:36 smithi173 ceph-mon[33421]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:37.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:37 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1930688843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:37.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:37 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1930688843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:37.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:37 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1930688843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:38.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:38 smithi173 ceph-mon[33421]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:38.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:38 smithi079 ceph-mon[29577]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:38.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:38 smithi079 ceph-mon[40839]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:40.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:40 smithi173 ceph-mon[33421]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:40.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:40 smithi079 ceph-mon[29577]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:40.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:40 smithi079 ceph-mon[40839]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:41.593 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:41.918 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:41.918 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:41.918 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:42.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:42 smithi173 ceph-mon[33421]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:42.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:42 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1364776363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:42.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:11:42 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:11:42.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:42 smithi079 ceph-mon[29577]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:42.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:42 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1364776363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:42.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:42 smithi079 ceph-mon[40839]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:42.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:42 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1364776363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:43.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:43 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:43.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:43 smithi173 ceph-mon[33421]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:43 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:43 smithi079 ceph-mon[29577]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:43.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:43 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:43.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:43 smithi079 ceph-mon[40839]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:43.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:11:43 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:11:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:11:46.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:46 smithi173 ceph-mon[33421]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:46.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:46 smithi079 ceph-mon[29577]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:46.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:46 smithi079 ceph-mon[40839]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:46.920 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:47.240 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:47.240 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:47.240 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:47.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:47 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:47.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:47 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1070946388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:47.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:47 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:47.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:47 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1070946388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:47.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:47 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:11:47.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:47 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1070946388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:48.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:48 smithi173 ceph-mon[33421]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:48.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:48 smithi079 ceph-mon[29577]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:48.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:48 smithi079 ceph-mon[40839]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:50.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:50 smithi173 ceph-mon[33421]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:50.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:50 smithi079 ceph-mon[29577]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:50.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:50 smithi079 ceph-mon[40839]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:51.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:51 smithi079 ceph-mon[29577]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:51.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:51 smithi079 ceph-mon[40839]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:52.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:51 smithi173 ceph-mon[33421]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:52.242 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:52.572 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:52.572 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:52.572 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:52.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:52 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1882269907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:52.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:52 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1882269907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:53.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:52 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1882269907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:53.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:11:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:11:53.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:53 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:53.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:53 smithi079 ceph-mon[40839]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:53 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:53 smithi079 ceph-mon[29577]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:53.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:11:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:11:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:11:54.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:53 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:11:54.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:53 smithi173 ceph-mon[33421]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:56.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:56 smithi173 ceph-mon[33421]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:56.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:56 smithi079 ceph-mon[29577]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:56.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:56 smithi079 ceph-mon[40839]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:11:57.574 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:11:57.903 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:11:57.903 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:11:57.903 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:11:58.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:58 smithi173 ceph-mon[33421]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:58.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:11:58 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/4223197888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:58.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:58 smithi079 ceph-mon[29577]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:58.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:11:58 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4223197888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:11:58.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:58 smithi079 ceph-mon[40839]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:11:58.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:11:58 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/4223197888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:00.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:00 smithi173 ceph-mon[33421]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:00.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:00 smithi079 ceph-mon[29577]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:00.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:00 smithi079 ceph-mon[40839]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:01.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:01 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:12:01.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:01 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:12:01.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:01 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:12:02.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:12:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:12:02.760 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:02 smithi173 ceph-mon[33421]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:02.760 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:02.761 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:12:02.761 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:12:02.761 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:12:02.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[40839]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:12:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:12:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:12:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[29577]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:12:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:12:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:12:02.905 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:03.232 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:03.233 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:03.233 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:03.510 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:03 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3672824519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:03.510 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:03 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3672824519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:03.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:03 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3672824519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:03.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:12:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:12:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:12:04.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:04 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:04.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:04 smithi173 ceph-mon[33421]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:04.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:04 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:04.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:04 smithi079 ceph-mon[40839]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:04.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:04 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:04.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:04 smithi079 ceph-mon[29577]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:05.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:05 smithi173 ceph-mon[33421]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:05.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:05 smithi079 ceph-mon[29577]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:05.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:05 smithi079 ceph-mon[40839]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:08.234 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:08.561 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:08.561 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:08.561 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:08.562 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:08 smithi079 ceph-mon[29577]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:08.562 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:08 smithi079 ceph-mon[40839]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:08.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:08 smithi173 ceph-mon[33421]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:09.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:09 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/336760674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:09.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:09 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/336760674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:09.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:09 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/336760674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:10.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:10 smithi173 ceph-mon[33421]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:10.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:10 smithi079 ceph-mon[29577]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:10.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:10 smithi079 ceph-mon[40839]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:12.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:12 smithi173 ceph-mon[33421]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:12.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:12:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:12:12.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:12 smithi079 ceph-mon[29577]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:12.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:12 smithi079 ceph-mon[40839]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:13.563 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:13.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:13 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:13.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:13 smithi173 ceph-mon[33421]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:13.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:13 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:13.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:13 smithi079 ceph-mon[29577]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:13.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:12:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:12:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:12:13.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:13 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:13.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:13 smithi079 ceph-mon[40839]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:13.889 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:13.889 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:13.889 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:14.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:14 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3230220423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:14.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:14 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3230220423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:14.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:14 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3230220423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:15.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:15 smithi173 ceph-mon[33421]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:15.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:15 smithi079 ceph-mon[29577]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:15.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:15 smithi079 ceph-mon[40839]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:17.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:16 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:17.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:16 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:17.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:16 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:18.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:17 smithi173 ceph-mon[33421]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:18.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:17 smithi079 ceph-mon[29577]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:18.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:17 smithi079 ceph-mon[40839]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:18.891 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:19.214 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:19.214 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:19.214 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:19.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:19 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/818443079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:19.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:19 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/818443079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:19.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:19 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/818443079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:20.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:20 smithi079 ceph-mon[29577]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:20.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:20 smithi079 ceph-mon[40839]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:20.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:20 smithi173 ceph-mon[33421]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:22.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:22 smithi173 ceph-mon[33421]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:22.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:12:22 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:12:22.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:22 smithi079 ceph-mon[29577]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:22.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:22 smithi079 ceph-mon[40839]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:23.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:12:23 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:12:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:12:24.216 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:24.545 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:24.545 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:24.545 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:24.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:24 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:24.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:24 smithi173 ceph-mon[33421]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:24 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:24 smithi079 ceph-mon[29577]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:24 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:24 smithi079 ceph-mon[40839]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:25.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:25 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/476433752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:25.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:25 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/476433752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:25.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:25 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/476433752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:26.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:26 smithi173 ceph-mon[33421]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:26.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:26 smithi079 ceph-mon[29577]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:26.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:26 smithi079 ceph-mon[40839]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:27.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:27 smithi173 ceph-mon[33421]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:27.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:27 smithi079 ceph-mon[29577]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:27.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:27 smithi079 ceph-mon[40839]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:29.874 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:29.874 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:29.874 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:30.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:30 smithi173 ceph-mon[33421]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:30.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:30 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/901087609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:30.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:30 smithi079 ceph-mon[29577]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:30.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:30 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/901087609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:30.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:30 smithi079 ceph-mon[40839]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:30.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:30 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/901087609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:32.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:32 smithi173 ceph-mon[33421]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:32.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:32.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:12:32 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:12:32.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:32 smithi079 ceph-mon[29577]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:32.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:32.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:32 smithi079 ceph-mon[40839]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:32.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:33.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:12:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:12:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:12:34.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:34 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:34.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:34 smithi173 ceph-mon[33421]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:34.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:34 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:34.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:34 smithi079 ceph-mon[29577]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:34.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:34 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:34.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:34 smithi079 ceph-mon[40839]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:34.875 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:35.197 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:35.197 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:35.198 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:35.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:35 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1619131103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:35.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:35 smithi173 ceph-mon[33421]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:35.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:35 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1619131103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:35.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:35 smithi079 ceph-mon[29577]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:35.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:35 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1619131103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:35.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:35 smithi079 ceph-mon[40839]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:38.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:38 smithi173 ceph-mon[33421]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:38.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:38 smithi079 ceph-mon[29577]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:38.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:38 smithi079 ceph-mon[40839]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:40.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:39 smithi173 ceph-mon[33421]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:40.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:39 smithi079 ceph-mon[40839]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:40.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:39 smithi079 ceph-mon[29577]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:40.199 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:40.525 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:40.525 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:40.525 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:40.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:40 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1146874628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:40.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:40 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1146874628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:41.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:40 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1146874628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:42.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:41 smithi173 ceph-mon[33421]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:42.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:41 smithi079 ceph-mon[29577]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:42.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:41 smithi079 ceph-mon[40839]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:43.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:12:42 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:12:43.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:12:43 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:12:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:12:44.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:44 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:44.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:44 smithi173 ceph-mon[33421]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:44.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:44 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:44.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:44 smithi079 ceph-mon[29577]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:44.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:44 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:44.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:44 smithi079 ceph-mon[40839]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:45.526 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:45.849 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:45.849 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:45.849 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:46.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:46 smithi173 ceph-mon[33421]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:46.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:46 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/267930061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:46.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:46 smithi079 ceph-mon[29577]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:46 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/267930061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:46.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:46 smithi079 ceph-mon[40839]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:46.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:46 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/267930061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:47.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:47 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:47.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:47 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:47.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:47 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:12:48.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:48 smithi173 ceph-mon[33421]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:48.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:48 smithi079 ceph-mon[29577]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:48.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:48 smithi079 ceph-mon[40839]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:49.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:49 smithi173 ceph-mon[33421]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:49.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:49 smithi079 ceph-mon[29577]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:49.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:49 smithi079 ceph-mon[40839]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:50.850 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:51.175 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:51.176 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:51.176 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:51.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:51 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3609200626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:51.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:51 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3609200626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:51.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:51 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3609200626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:52.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:52 smithi173 ceph-mon[33421]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:52.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:52 smithi079 ceph-mon[29577]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:52.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:52 smithi079 ceph-mon[40839]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:53.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:12:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:12:53.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:12:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:12:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:12:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:54 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:54.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:54 smithi173 ceph-mon[33421]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:54.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:54 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:54.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:54 smithi079 ceph-mon[40839]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:54 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:12:54.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:54 smithi079 ceph-mon[29577]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:55.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:55 smithi173 ceph-mon[33421]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:55.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:55 smithi079 ceph-mon[29577]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:55.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:55 smithi079 ceph-mon[40839]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:12:56.177 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:12:56.501 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:12:56.501 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:12:56.502 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:12:56.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1331913524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:56.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1331913524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:57.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1331913524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:12:57.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:12:57 smithi079 ceph-mon[40839]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:57.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:12:57 smithi079 ceph-mon[29577]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:12:58.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:12:57 smithi173 ceph-mon[33421]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:00.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:00 smithi173 ceph-mon[33421]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:00.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:00 smithi079 ceph-mon[29577]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:00.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:00 smithi079 ceph-mon[40839]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:01.503 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:13:01.832 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:13:01.832 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:13:01.832 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:01.833 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:01 smithi079 ceph-mon[29577]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:01.833 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:01 smithi079 ceph-mon[40839]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:02.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:01 smithi173 ceph-mon[33421]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:02.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:02.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1206244934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1206244934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:13:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:03.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:13:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:13:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:13:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:02 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1206244934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:13:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:13:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:03.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:03 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:03.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:03 smithi079 ceph-mon[29577]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:03.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:13:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:13:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:13:03.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:03 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:03.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:03 smithi079 ceph-mon[40839]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:04.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:03 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:04.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:03 smithi173 ceph-mon[33421]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:06.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:06 smithi173 ceph-mon[33421]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:06.834 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:13:06.835 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:06 smithi079 ceph-mon[29577]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:06.835 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:06 smithi079 ceph-mon[40839]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:13:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:13:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:07.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:07 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3371564389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:07.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:07 smithi173 ceph-mon[33421]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:07.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:07 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3371564389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:07.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:07 smithi079 ceph-mon[29577]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:07.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:07 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3371564389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:07.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:07 smithi079 ceph-mon[40839]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:09.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:09 smithi079 ceph-mon[29577]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:09.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:09 smithi079 ceph-mon[40839]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:10.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:09 smithi173 ceph-mon[33421]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:11.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:11 smithi079 ceph-mon[29577]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:11.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:11 smithi079 ceph-mon[40839]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:12.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:11 smithi173 ceph-mon[33421]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:12.160 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:13:12.498 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== 2024-09-17T09:13:12.498 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== ']' 2024-09-17T09:13:12.498 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:12.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:12 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1760262648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:12.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:12 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1760262648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:13.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:12 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1760262648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:13.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:13:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:13:13.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:13 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:13.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:13 smithi079 ceph-mon[29577]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:13.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:13 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:13.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:13 smithi079 ceph-mon[40839]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:13.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:13:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:13:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:13:14.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:13 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:14.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:13 smithi173 ceph-mon[33421]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:15.865 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:15 smithi079 ceph-mon[29577]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T09:13:15.865 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:15 smithi079 ceph-mon[40839]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T09:13:16.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:15 smithi173 ceph-mon[33421]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T09:13:17.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:16 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:17.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:16 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:17.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:16 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:17.502 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.0 2024-09-17T09:13:17.828 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for osd.1 2024-09-17T09:13:17.829 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBVR+lmtwMwDRAAH946ZxrDxpxDJ8n//6srvA== 2024-09-17T09:13:17.829 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAoRulmf3uDEBAANILU0jKEqJTr2o5LwY7lTA== == AQBVR+lmtwMwDRAAH946ZxrDxpxDJ8n//6srvA== ']' 2024-09-17T09:13:17.829 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:13:17.829 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for osd.1' 2024-09-17T09:13:17.830 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:17 smithi079 ceph-mon[29577]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:17.830 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:17 smithi079 ceph-mon[40839]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:17.831 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.1 2024-09-17T09:13:18.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:17 smithi173 ceph-mon[33421]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:18.141 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== 2024-09-17T09:13:18.141 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== 2024-09-17T09:13:18.142 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key osd.1 2024-09-17T09:13:18.405 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key osd.1 on host 'smithi079' 2024-09-17T09:13:18.419 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== == AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== ']' 2024-09-17T09:13:18.419 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:18.928 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2742899257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:18.928 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/159954539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: Schedule rotate-key daemon osd.1 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]': finished 2024-09-17T09:13:18.929 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2742899257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/159954539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: Schedule rotate-key daemon osd.1 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2024-09-17T09:13:18.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:18 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]': finished 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2742899257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/159954539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: Schedule rotate-key daemon osd.1 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:19.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2024-09-17T09:13:19.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:18 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]': finished 2024-09-17T09:13:20.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:19 smithi173 ceph-mon[33421]: Rotating authentication key for osd.1 2024-09-17T09:13:20.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:19 smithi173 ceph-mon[33421]: Reconfiguring daemon osd.1 on smithi079 2024-09-17T09:13:20.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:19 smithi173 ceph-mon[33421]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T09:13:20.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:19 smithi079 ceph-mon[29577]: Rotating authentication key for osd.1 2024-09-17T09:13:20.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:19 smithi079 ceph-mon[29577]: Reconfiguring daemon osd.1 on smithi079 2024-09-17T09:13:20.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:19 smithi079 ceph-mon[29577]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T09:13:20.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:19 smithi079 ceph-mon[40839]: Rotating authentication key for osd.1 2024-09-17T09:13:20.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:19 smithi079 ceph-mon[40839]: Reconfiguring daemon osd.1 on smithi079 2024-09-17T09:13:20.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:19 smithi079 ceph-mon[40839]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T09:13:21.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:20 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:20 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:20 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:20 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:20 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:20 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:20 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:20 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:20 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:20 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:20 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:21.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:20 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:22.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:21 smithi079 ceph-mon[29577]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T09:13:22.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:21 smithi079 ceph-mon[40839]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T09:13:22.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:21 smithi173 ceph-mon[33421]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T09:13:23.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:13:22 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:13:23.422 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.1 2024-09-17T09:13:23.749 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== 2024-09-17T09:13:23.749 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== == AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== ']' 2024-09-17T09:13:23.750 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:23.750 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:23 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:23.750 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:23 smithi079 ceph-mon[29577]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:23.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:23 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:23.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:23 smithi079 ceph-mon[40839]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:23.751 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:13:23 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:13:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:13:24.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:23 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:24.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:23 smithi173 ceph-mon[33421]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:24 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2134625866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:24 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2134625866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:25.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:24 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2134625866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:25.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:25 smithi079 ceph-mon[29577]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:25.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:25 smithi079 ceph-mon[40839]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:26.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:25 smithi173 ceph-mon[33421]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:27.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:27 smithi079 ceph-mon[29577]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:27.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:27 smithi079 ceph-mon[40839]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:28.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:27 smithi173 ceph-mon[33421]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:28.752 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.1 2024-09-17T09:13:29.074 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== 2024-09-17T09:13:29.074 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== == AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== ']' 2024-09-17T09:13:29.074 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:29.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:29 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3924645746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:29.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:29 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3924645746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:29.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:29 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3924645746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:30.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:30 smithi173 ceph-mon[33421]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:30.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:30 smithi079 ceph-mon[29577]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:30.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:30 smithi079 ceph-mon[40839]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:31.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:31 smithi079 ceph-mon[29577]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:31.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:31 smithi079 ceph-mon[40839]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:32.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:31 smithi173 ceph-mon[33421]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:32.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:32.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:33.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:33.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:13:32 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:13:33.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:33 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:33.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:33 smithi079 ceph-mon[40839]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:33.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:33 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:33.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:33 smithi079 ceph-mon[29577]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:33.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:13:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:13:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:13:34.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:33 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:34.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:33 smithi173 ceph-mon[33421]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:34.076 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.1 2024-09-17T09:13:34.403 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for osd.2 2024-09-17T09:13:34.404 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAuSOlm4hwVHRAAppRNDGDuNlQiC4shaYCpvg== 2024-09-17T09:13:34.404 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQA+RulmKFO+OBAA5Ba0kd+GdOIIo0QyGRNqWQ== == AQAuSOlm4hwVHRAAppRNDGDuNlQiC4shaYCpvg== ']' 2024-09-17T09:13:34.404 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:13:34.404 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for osd.2' 2024-09-17T09:13:34.404 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.2 2024-09-17T09:13:34.716 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== 2024-09-17T09:13:34.716 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== 2024-09-17T09:13:34.716 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key osd.2 2024-09-17T09:13:34.981 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key osd.2 on host 'smithi079' 2024-09-17T09:13:34.993 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== == AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== ']' 2024-09-17T09:13:34.993 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:35.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1570585780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:35.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/153798515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:35.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:35.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:13:35.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1570585780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:35.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/153798515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:35.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:35.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:35.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:13:35.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:35 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1570585780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-17T09:13:35.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:35 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/153798515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:35.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:35.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:35.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: Schedule rotate-key daemon osd.2 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: Rotating authentication key for osd.2 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]': finished 2024-09-17T09:13:36.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: Reconfiguring daemon osd.2 on smithi079 2024-09-17T09:13:36.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:36 smithi173 ceph-mon[33421]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:36.590 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: Schedule rotate-key daemon osd.2 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: Rotating authentication key for osd.2 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]': finished 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: Reconfiguring daemon osd.2 on smithi079 2024-09-17T09:13:36.591 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[29577]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: Schedule rotate-key daemon osd.2 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: Rotating authentication key for osd.2 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]': finished 2024-09-17T09:13:36.592 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: Reconfiguring daemon osd.2 on smithi079 2024-09-17T09:13:36.593 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:36 smithi079 ceph-mon[40839]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:37.924 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:37.924 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:37.924 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:37.924 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:37.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[29577]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:37.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:37.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:37.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:37.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:37.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:37 smithi079 ceph-mon[40839]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:38.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:37 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:37 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:37 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:37 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:13:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:37 smithi173 ceph-mon[33421]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:39.996 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:39 smithi079 ceph-mon[40839]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:39.996 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:39 smithi079 ceph-mon[29577]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:39.996 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.2 2024-09-17T09:13:40.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:39 smithi173 ceph-mon[33421]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:40.322 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== 2024-09-17T09:13:40.322 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== == AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== ']' 2024-09-17T09:13:40.322 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:41.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:40 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1958805507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:41.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:40 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1958805507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:41.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:40 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1958805507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:42.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:41 smithi173 ceph-mon[33421]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:42.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:41 smithi079 ceph-mon[29577]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:42.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:41 smithi079 ceph-mon[40839]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:43.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:13:42 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:13:43.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:43 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:43.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:43 smithi079 ceph-mon[29577]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:43.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:43 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:43.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:43 smithi079 ceph-mon[40839]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:43.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:13:43 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:13:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:13:44.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:43 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:44.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:43 smithi173 ceph-mon[33421]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:45.324 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.2 2024-09-17T09:13:45.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:45 smithi079 ceph-mon[29577]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:45.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:45 smithi079 ceph-mon[40839]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:45.653 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== 2024-09-17T09:13:45.653 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== == AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== ']' 2024-09-17T09:13:45.653 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:46.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:45 smithi173 ceph-mon[33421]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:46.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:46 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1772298672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:46.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:46 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1772298672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:46.907 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:46 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1772298672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:47.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:47 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:47 smithi079 ceph-mon[29577]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:47.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:47 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:47.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:47 smithi079 ceph-mon[40839]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:48.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:47 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:13:48.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:47 smithi173 ceph-mon[33421]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:49.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:49 smithi079 ceph-mon[29577]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:49.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:49 smithi079 ceph-mon[40839]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:50.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:49 smithi173 ceph-mon[33421]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:50.655 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.2 2024-09-17T09:13:50.979 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== 2024-09-17T09:13:50.979 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== == AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== ']' 2024-09-17T09:13:50.979 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:51.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:51 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3178678445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:51.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:51 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3178678445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:51 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3178678445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:52.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:52 smithi079 ceph-mon[29577]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:52.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:52 smithi079 ceph-mon[40839]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:52.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:52 smithi173 ceph-mon[33421]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:53.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:13:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:13:53.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:13:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:13:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:13:53.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:53 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:53.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:53 smithi079 ceph-mon[40839]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:53 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:53 smithi079 ceph-mon[29577]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:54.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:53 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:13:54.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:53 smithi173 ceph-mon[33421]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:55.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:55 smithi079 ceph-mon[29577]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:55.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:55 smithi079 ceph-mon[40839]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:55.980 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.2 2024-09-17T09:13:56.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:55 smithi173 ceph-mon[33421]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:13:56.306 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== 2024-09-17T09:13:56.306 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== == AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== ']' 2024-09-17T09:13:56.306 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:13:56.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/766860521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:56.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/766860521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:57.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/766860521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:13:57.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:57 smithi079 ceph-mon[29577]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:57.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:57 smithi079 ceph-mon[40839]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:13:58.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:57 smithi173 ceph-mon[33421]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:00.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:13:59 smithi173 ceph-mon[33421]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:00.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:13:59 smithi079 ceph-mon[29577]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:00.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:13:59 smithi079 ceph-mon[40839]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:01.308 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.2 2024-09-17T09:14:01.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:01 smithi079 ceph-mon[29577]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:01.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:01 smithi079 ceph-mon[40839]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:01.637 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for osd.3 2024-09-17T09:14:01.638 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQA/SOlmgfQjEhAADaUHXiW3hie9NnsX4bI6iA== 2024-09-17T09:14:01.638 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBVRulmXLTOHhAANg8hbsX0EVFWiJUMwkCjrQ== == AQA/SOlmgfQjEhAADaUHXiW3hie9NnsX4bI6iA== ']' 2024-09-17T09:14:01.638 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:14:01.638 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for osd.3' 2024-09-17T09:14:01.638 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.3 2024-09-17T09:14:01.952 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== 2024-09-17T09:14:01.953 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== 2024-09-17T09:14:01.953 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key osd.3 2024-09-17T09:14:02.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:01 smithi173 ceph-mon[33421]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:02.213 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key osd.3 on host 'smithi079' 2024-09-17T09:14:02.225 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== == AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== ']' 2024-09-17T09:14:02.225 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:02.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2813838085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:14:02.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:02.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3004796296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:14:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: Schedule rotate-key daemon osd.3 2024-09-17T09:14:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:14:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2813838085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:14:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:02.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3004796296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:02.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:14:02.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: Schedule rotate-key daemon osd.3 2024-09-17T09:14:02.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:02.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:02.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:14:02.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:02.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2813838085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-17T09:14:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3004796296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:14:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: Schedule rotate-key daemon osd.3 2024-09-17T09:14:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:14:03.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:03.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:03.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:14:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:14:03.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:14:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[40839]: Rotating authentication key for osd.3 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]': finished 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[40839]: Reconfiguring daemon osd.3 on smithi079 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[40839]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:03.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:14:03.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:14:03.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:03.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[29577]: Rotating authentication key for osd.3 2024-09-17T09:14:03.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2024-09-17T09:14:03.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]': finished 2024-09-17T09:14:03.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[29577]: Reconfiguring daemon osd.3 on smithi079 2024-09-17T09:14:03.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:03 smithi079 ceph-mon[29577]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:04.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:03 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:04.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:03 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:14:04.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:03 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:14:04.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:03 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:04.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:03 smithi173 ceph-mon[33421]: Rotating authentication key for osd.3 2024-09-17T09:14:04.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:03 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2024-09-17T09:14:04.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:03 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]': finished 2024-09-17T09:14:04.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:03 smithi173 ceph-mon[33421]: Reconfiguring daemon osd.3 on smithi079 2024-09-17T09:14:04.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:03 smithi173 ceph-mon[33421]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:05.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:05 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:05 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:05 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:05 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:05 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:05 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:05 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:05 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:05 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:05 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:05 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:05.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:05 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:06.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:06 smithi079 ceph-mon[29577]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:06.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:06 smithi079 ceph-mon[40839]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:06.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:06 smithi173 ceph-mon[33421]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:07.227 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.3 2024-09-17T09:14:07.559 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== 2024-09-17T09:14:07.559 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== == AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== ']' 2024-09-17T09:14:07.559 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:07.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:07 smithi079 ceph-mon[29577]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:07.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:07 smithi079 ceph-mon[40839]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:08.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:07 smithi173 ceph-mon[33421]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:08.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:08 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3604082486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:08 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3604082486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:09.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:08 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3604082486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:09.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:09 smithi079 ceph-mon[29577]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:09.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:09 smithi079 ceph-mon[40839]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:10.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:09 smithi173 ceph-mon[33421]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:11.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:11 smithi079 ceph-mon[29577]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:11.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:11 smithi079 ceph-mon[40839]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:12.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:11 smithi173 ceph-mon[33421]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:12.561 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.3 2024-09-17T09:14:12.883 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== 2024-09-17T09:14:12.883 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== == AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== ']' 2024-09-17T09:14:12.883 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:13.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:12 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3524644266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:13.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:14:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:14:13.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:12 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3524644266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:13.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:12 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3524644266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:13.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:14:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:14:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:14:14.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:13 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:14.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:13 smithi173 ceph-mon[33421]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:14.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:13 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:14.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:13 smithi079 ceph-mon[29577]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:14.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:13 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:14.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:13 smithi079 ceph-mon[40839]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:15.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:15 smithi079 ceph-mon[29577]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:15.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:15 smithi079 ceph-mon[40839]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:16.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:15 smithi173 ceph-mon[33421]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:17.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:16 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:17.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:16 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:17.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:16 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:17.885 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.3 2024-09-17T09:14:18.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:17 smithi173 ceph-mon[33421]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:18.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:17 smithi079 ceph-mon[29577]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:18.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:17 smithi079 ceph-mon[40839]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== 2024-09-17T09:14:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== == AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== ']' 2024-09-17T09:14:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:19.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:18 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2187184829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:19.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:18 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2187184829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:19.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:18 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2187184829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:20.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:20 smithi079 ceph-mon[29577]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:20.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:20 smithi079 ceph-mon[40839]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:20.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:20 smithi173 ceph-mon[33421]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:21.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:21 smithi079 ceph-mon[29577]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:21.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:21 smithi079 ceph-mon[40839]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:22.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:21 smithi173 ceph-mon[33421]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:23.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:14:22 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:14:23.218 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.3 2024-09-17T09:14:23.545 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for osd.4 2024-09-17T09:14:23.545 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBaSOlm/sSWMhAAQ+ppMubDk8JpayrwsSomrQ== 2024-09-17T09:14:23.546 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQBsRulmv0EvKRAAlwVQE+eL0ouuSeszgi6qng== == AQBaSOlm/sSWMhAAQ+ppMubDk8JpayrwsSomrQ== ']' 2024-09-17T09:14:23.546 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:14:23.546 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for osd.4' 2024-09-17T09:14:23.546 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.4 2024-09-17T09:14:23.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:14:23 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:14:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:14:23.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:23 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:23.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:23 smithi079 ceph-mon[29577]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:23.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:23 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2241709565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:23.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:23 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:23.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:23 smithi079 ceph-mon[40839]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:23.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:23 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2241709565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:23.869 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== 2024-09-17T09:14:23.869 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== 2024-09-17T09:14:23.870 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key osd.4 2024-09-17T09:14:24.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:23 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:24.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:23 smithi173 ceph-mon[33421]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:24.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:23 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2241709565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-17T09:14:24.130 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key osd.4 on host 'smithi173' 2024-09-17T09:14:24.142 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== == AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== ']' 2024-09-17T09:14:24.143 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/515621418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: Schedule rotate-key daemon osd.4 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:24.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: Rotating authentication key for osd.4 2024-09-17T09:14:24.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2024-09-17T09:14:24.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]': finished 2024-09-17T09:14:24.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:24 smithi173 ceph-mon[33421]: Reconfiguring daemon osd.4 on smithi173 2024-09-17T09:14:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/515621418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: Schedule rotate-key daemon osd.4 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: Rotating authentication key for osd.4 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]': finished 2024-09-17T09:14:24.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[29577]: Reconfiguring daemon osd.4 on smithi173 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/515621418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: Schedule rotate-key daemon osd.4 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: Rotating authentication key for osd.4 2024-09-17T09:14:24.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2024-09-17T09:14:24.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]': finished 2024-09-17T09:14:24.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:24 smithi079 ceph-mon[40839]: Reconfiguring daemon osd.4 on smithi173 2024-09-17T09:14:25.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:25 smithi079 ceph-mon[29577]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:25.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:25 smithi079 ceph-mon[40839]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:26.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:25 smithi173 ceph-mon[33421]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:27.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:26 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:26 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:26 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:26 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:26 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:26 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:26 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:26 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:26 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:26 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:26 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:27.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:26 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:28.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:27 smithi173 ceph-mon[33421]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:28.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:27 smithi079 ceph-mon[29577]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:28.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:27 smithi079 ceph-mon[40839]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:29.145 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.4 2024-09-17T09:14:29.471 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== 2024-09-17T09:14:29.471 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== == AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== ']' 2024-09-17T09:14:29.471 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:29.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:29 smithi079 ceph-mon[29577]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:29.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:29 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3186498287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:29.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:29 smithi079 ceph-mon[40839]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:29.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:29 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3186498287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:30.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:29 smithi173 ceph-mon[33421]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:30.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:29 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3186498287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:31.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:31 smithi079 ceph-mon[29577]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:31.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:31 smithi079 ceph-mon[40839]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:32.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:31 smithi173 ceph-mon[33421]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:32.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:32 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:32.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:32 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:33.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:32 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:33.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:14:32 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:14:33.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:14:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:14:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:14:33.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:33 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:33.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:33 smithi079 ceph-mon[40839]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:33.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:33 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:33.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:33 smithi079 ceph-mon[29577]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:34.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:33 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:34.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:33 smithi173 ceph-mon[33421]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:34.473 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.4 2024-09-17T09:14:34.794 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== 2024-09-17T09:14:34.795 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== == AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== ']' 2024-09-17T09:14:34.795 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:35.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:34 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3934503377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:35.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:34 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3934503377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:35.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:34 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3934503377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:36.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:35 smithi079 ceph-mon[29577]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:36.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:35 smithi079 ceph-mon[40839]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:36.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:35 smithi173 ceph-mon[33421]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:37.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:37 smithi079 ceph-mon[40839]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:37.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:37 smithi079 ceph-mon[29577]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:38.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:37 smithi173 ceph-mon[33421]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:39.796 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.4 2024-09-17T09:14:39.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:39 smithi079 ceph-mon[29577]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:39.797 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:39 smithi079 ceph-mon[40839]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:40.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:39 smithi173 ceph-mon[33421]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:40.120 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== 2024-09-17T09:14:40.120 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== == AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== ']' 2024-09-17T09:14:40.120 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:40.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:40 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1308608752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:40.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:40 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1308608752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:41.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:40 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1308608752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:42.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:41 smithi173 ceph-mon[33421]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:42.361 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:41 smithi079 ceph-mon[29577]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:42.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:41 smithi079 ceph-mon[40839]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:43.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:14:42 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:14:43.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:14:43 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:14:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:14:43.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:43 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:43.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:43 smithi079 ceph-mon[29577]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:43.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:43 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:43.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:43 smithi079 ceph-mon[40839]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:44.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:43 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:44.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:43 smithi173 ceph-mon[33421]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:45.122 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.4 2024-09-17T09:14:45.447 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for osd.5 2024-09-17T09:14:45.448 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQBwSOlm0lTEDBAAulWRbJ6RAz/ZAHPSbFKa2Q== 2024-09-17T09:14:45.448 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCCRulmgFJeJBAAmvYSWXH3yDHPdYb3frHSBA== == AQBwSOlm0lTEDBAAulWRbJ6RAz/ZAHPSbFKa2Q== ']' 2024-09-17T09:14:45.448 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:14:45.448 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for osd.5' 2024-09-17T09:14:45.448 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.5 2024-09-17T09:14:45.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:45 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1424895058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:45.760 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== 2024-09-17T09:14:45.760 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== 2024-09-17T09:14:45.760 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key osd.5 2024-09-17T09:14:45.761 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:45 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1424895058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:45.761 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:45 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1424895058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-17T09:14:46.027 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key osd.5 on host 'smithi173' 2024-09-17T09:14:46.039 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== == AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== ']' 2024-09-17T09:14:46.039 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:46.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:46.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2506449753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:14:46.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:46.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:46.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:14:46.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:14:46.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:14:46.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:46.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2024-09-17T09:14:46.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:46 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]': finished 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2506449753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]': finished 2024-09-17T09:14:46.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:14:46.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2506449753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:14:46.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:46.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:46.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:14:46.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:14:46.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:14:46.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:46.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2024-09-17T09:14:46.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:46 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]': finished 2024-09-17T09:14:47.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:47 smithi173 ceph-mon[33421]: from='client.15195 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.5", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:14:47.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:47 smithi173 ceph-mon[33421]: Schedule rotate-key daemon osd.5 2024-09-17T09:14:47.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:47 smithi173 ceph-mon[33421]: Rotating authentication key for osd.5 2024-09-17T09:14:47.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:47 smithi173 ceph-mon[33421]: Reconfiguring daemon osd.5 on smithi173 2024-09-17T09:14:47.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:47 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:47.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:47 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:47.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:47 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[29577]: from='client.15195 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.5", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:14:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[29577]: Schedule rotate-key daemon osd.5 2024-09-17T09:14:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[29577]: Rotating authentication key for osd.5 2024-09-17T09:14:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[29577]: Reconfiguring daemon osd.5 on smithi173 2024-09-17T09:14:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:47.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[40839]: from='client.15195 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.5", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:14:47.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[40839]: Schedule rotate-key daemon osd.5 2024-09-17T09:14:47.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[40839]: Rotating authentication key for osd.5 2024-09-17T09:14:47.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[40839]: Reconfiguring daemon osd.5 on smithi173 2024-09-17T09:14:47.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:14:47.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:47.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:47 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:48.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:48 smithi173 ceph-mon[33421]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:48.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:48 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:48.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:48 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:48.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:48 smithi079 ceph-mon[29577]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:48.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:48 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:48.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:48 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:48.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:48 smithi079 ceph-mon[40839]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:48.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:48 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:48.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:48 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:14:50.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:50 smithi173 ceph-mon[33421]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:50.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:50 smithi079 ceph-mon[29577]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:50.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:50 smithi079 ceph-mon[40839]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:51.044 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.5 2024-09-17T09:14:51.368 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== 2024-09-17T09:14:51.368 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== == AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== ']' 2024-09-17T09:14:51.368 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:51.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:51 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2143193546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:14:51.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:51 smithi173 ceph-mon[33421]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:51.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:51 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2143193546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:14:51.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:51 smithi079 ceph-mon[29577]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:51.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:51 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2143193546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:14:51.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:51 smithi079 ceph-mon[40839]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:53.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:14:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:14:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:53 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:53 smithi079 ceph-mon[29577]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:53.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:53 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:53.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:53 smithi079 ceph-mon[40839]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:53.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:14:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:14:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:14:54.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:53 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:14:54.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:53 smithi173 ceph-mon[33421]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:55.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:55 smithi079 ceph-mon[29577]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:55.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:55 smithi079 ceph-mon[40839]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:56.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:55 smithi173 ceph-mon[33421]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:56.370 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.5 2024-09-17T09:14:56.698 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== 2024-09-17T09:14:56.698 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== == AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== ']' 2024-09-17T09:14:56.698 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:14:57.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2036273636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:14:57.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2036273636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:14:57.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2036273636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:14:58.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:57 smithi173 ceph-mon[33421]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:58.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:57 smithi079 ceph-mon[29577]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:58.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:57 smithi079 ceph-mon[40839]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:14:59.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:14:59 smithi079 ceph-mon[29577]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:14:59.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:14:59 smithi079 ceph-mon[40839]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:15:00.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:14:59 smithi173 ceph-mon[33421]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T09:15:01.699 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.5 2024-09-17T09:15:01.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:01 smithi079 ceph-mon[29577]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:01.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:01 smithi079 ceph-mon[40839]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:02.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:01 smithi173 ceph-mon[33421]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:02.024 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== 2024-09-17T09:15:02.024 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== == AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== ']' 2024-09-17T09:15:02.024 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:02 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:02.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:02 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3383962368' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:15:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:02 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:02.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:02 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3383962368' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:15:03.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:02 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:03.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:02 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3383962368' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:15:03.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:15:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:15:03.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:03 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:03.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:03 smithi079 ceph-mon[40839]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:03.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:03 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:03.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:03 smithi079 ceph-mon[29577]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:03.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:15:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:15:04.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:03 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:04.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:03 smithi173 ceph-mon[33421]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:05.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:05 smithi079 ceph-mon[29577]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:05.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:05 smithi079 ceph-mon[40839]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:06.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:05 smithi173 ceph-mon[33421]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:07.026 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.5 2024-09-17T09:15:07.353 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for osd.6 2024-09-17T09:15:07.353 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCGSOlmiIyPBhAADPCRWgj9JBSFGbPnpopAcQ== 2024-09-17T09:15:07.353 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCaRulmS9rAEBAA1x7boEoxx2590VoECf265g== == AQCGSOlmiIyPBhAADPCRWgj9JBSFGbPnpopAcQ== ']' 2024-09-17T09:15:07.353 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:15:07.354 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for osd.6' 2024-09-17T09:15:07.354 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.6 2024-09-17T09:15:07.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:07 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/200079831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:15:07.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:07 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/200079831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:15:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== 2024-09-17T09:15:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== 2024-09-17T09:15:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key osd.6 2024-09-17T09:15:07.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:07 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/200079831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-17T09:15:07.937 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key osd.6 on host 'smithi173' 2024-09-17T09:15:07.948 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== == AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== ']' 2024-09-17T09:15:07.948 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/4260255586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2024-09-17T09:15:08.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:08 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]': finished 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/4260255586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2024-09-17T09:15:08.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]': finished 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/4260255586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2024-09-17T09:15:08.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:08 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]': finished 2024-09-17T09:15:09.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:09 smithi173 ceph-mon[33421]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.6", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:15:09.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:09 smithi173 ceph-mon[33421]: Schedule rotate-key daemon osd.6 2024-09-17T09:15:09.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:09 smithi173 ceph-mon[33421]: Rotating authentication key for osd.6 2024-09-17T09:15:09.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:09 smithi173 ceph-mon[33421]: Reconfiguring daemon osd.6 on smithi173 2024-09-17T09:15:09.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:09 smithi079 ceph-mon[29577]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.6", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:15:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:09 smithi079 ceph-mon[29577]: Schedule rotate-key daemon osd.6 2024-09-17T09:15:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:09 smithi079 ceph-mon[29577]: Rotating authentication key for osd.6 2024-09-17T09:15:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:09 smithi079 ceph-mon[29577]: Reconfiguring daemon osd.6 on smithi173 2024-09-17T09:15:09.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:09 smithi079 ceph-mon[40839]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.6", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:15:09.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:09 smithi079 ceph-mon[40839]: Schedule rotate-key daemon osd.6 2024-09-17T09:15:09.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:09 smithi079 ceph-mon[40839]: Rotating authentication key for osd.6 2024-09-17T09:15:09.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:09 smithi079 ceph-mon[40839]: Reconfiguring daemon osd.6 on smithi173 2024-09-17T09:15:10.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:10 smithi173 ceph-mon[33421]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:10.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:10 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:10 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:10 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:10 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[29577]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:10.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[40839]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:10.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:10.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:10 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:11.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:11 smithi079 ceph-mon[29577]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:11.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:11 smithi079 ceph-mon[40839]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:12.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:11 smithi173 ceph-mon[33421]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:12.950 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.6 2024-09-17T09:15:13.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:15:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:15:13.274 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== 2024-09-17T09:15:13.274 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== == AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== ']' 2024-09-17T09:15:13.275 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:13.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:13 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/203507473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:13.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:13 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/203507473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:13.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:15:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:15:13.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:13 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/203507473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:14.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:14 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:14.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:14 smithi079 ceph-mon[29577]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:14.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:14 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:14.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:14 smithi079 ceph-mon[40839]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:14.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:14 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:14.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:14 smithi173 ceph-mon[33421]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:15.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:15 smithi079 ceph-mon[29577]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:15.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:15 smithi079 ceph-mon[40839]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:16.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:15 smithi173 ceph-mon[33421]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:17.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:16 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:17.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:16 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:17.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:16 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:18.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:17 smithi173 ceph-mon[33421]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:18.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:17 smithi079 ceph-mon[29577]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:18.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:17 smithi079 ceph-mon[40839]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:18.276 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.6 2024-09-17T09:15:18.602 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== 2024-09-17T09:15:18.602 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== == AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== ']' 2024-09-17T09:15:18.603 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:18.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:18 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2761822133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:18.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:18 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2761822133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:19.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:18 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2761822133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:20.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:19 smithi079 ceph-mon[40839]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:20.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:19 smithi079 ceph-mon[29577]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:20.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:19 smithi173 ceph-mon[33421]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:21.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:21 smithi079 ceph-mon[29577]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:21.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:21 smithi079 ceph-mon[40839]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:22.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:21 smithi173 ceph-mon[33421]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:23.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:15:22 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:15:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.6 2024-09-17T09:15:23.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:23 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:23.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:23 smithi079 ceph-mon[29577]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:23.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:23 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:23.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:23 smithi079 ceph-mon[40839]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:23.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:23 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:15:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:15:23.927 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== 2024-09-17T09:15:23.927 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== == AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== ']' 2024-09-17T09:15:23.927 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:24.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:23 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:24.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:23 smithi173 ceph-mon[33421]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:24.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:24 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/253947872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:24.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:24 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/253947872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:25.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:24 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/253947872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:25.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:25 smithi079 ceph-mon[29577]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:25.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:25 smithi079 ceph-mon[40839]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:26.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:25 smithi173 ceph-mon[33421]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:27.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:27 smithi079 ceph-mon[29577]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:27.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:27 smithi079 ceph-mon[40839]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:28.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:27 smithi173 ceph-mon[33421]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:28.929 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.6 2024-09-17T09:15:29.254 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== 2024-09-17T09:15:29.254 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== == AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== ']' 2024-09-17T09:15:29.254 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:29.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:29 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/927007789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:29.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:29 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/927007789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:29.757 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:29 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/927007789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:30.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:30 smithi079 ceph-mon[40839]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:30.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:30 smithi079 ceph-mon[29577]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:30.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:30 smithi173 ceph-mon[33421]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:31 smithi173 ceph-mon[33421]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:32.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:31 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:32.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:31 smithi079 ceph-mon[40839]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:32.361 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:31 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:31 smithi079 ceph-mon[29577]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:32.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:31 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:33.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:15:32 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:15:33.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:33 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:33.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:33 smithi079 ceph-mon[40839]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:33.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:33 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:33.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:33 smithi079 ceph-mon[29577]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:33.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:33 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:15:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:15:34.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:33 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:34.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:33 smithi173 ceph-mon[33421]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:34.256 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.6 2024-09-17T09:15:34.583 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for osd.7 2024-09-17T09:15:34.583 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCcSOlmg4iwFBAAJqz092kWsVZdSL76Bn9KUA== 2024-09-17T09:15:34.584 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCyRulmwpLdNxAAhc61X9Bxeft5OhNPes5RvQ== == AQCcSOlmg4iwFBAAJqz092kWsVZdSL76Bn9KUA== ']' 2024-09-17T09:15:34.584 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:15:34.584 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for osd.7' 2024-09-17T09:15:34.584 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.7 2024-09-17T09:15:34.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:34 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1135391343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:34.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:34 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1135391343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:34.897 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== 2024-09-17T09:15:34.897 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== 2024-09-17T09:15:34.897 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key osd.7 2024-09-17T09:15:35.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:34 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1135391343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-17T09:15:35.160 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key osd.7 on host 'smithi173' 2024-09-17T09:15:35.172 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== == AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== ']' 2024-09-17T09:15:35.172 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:35.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1138559590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:35.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='client.25148 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:15:35.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: Schedule rotate-key daemon osd.7 2024-09-17T09:15:35.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:35.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:35.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:35.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:35.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:15:35.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:35.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: Rotating authentication key for osd.7 2024-09-17T09:15:35.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2024-09-17T09:15:35.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]': finished 2024-09-17T09:15:35.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: Reconfiguring daemon osd.7 on smithi173 2024-09-17T09:15:35.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:35 smithi173 ceph-mon[33421]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:36.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1138559590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:36.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='client.25148 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:15:36.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: Schedule rotate-key daemon osd.7 2024-09-17T09:15:36.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:36.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:36.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:36.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:36.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: Rotating authentication key for osd.7 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]': finished 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: Reconfiguring daemon osd.7 on smithi173 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[29577]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1138559590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='client.25148 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: Schedule rotate-key daemon osd.7 2024-09-17T09:15:36.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: Rotating authentication key for osd.7 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]': finished 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: Reconfiguring daemon osd.7 on smithi173 2024-09-17T09:15:36.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:35 smithi079 ceph-mon[40839]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:37.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:37.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:37.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:37.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:37.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[29577]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:37.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:37.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:37.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:37.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:37.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:37 smithi079 ceph-mon[40839]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:37 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:37 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:37 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:37 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:38.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:37 smithi173 ceph-mon[33421]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:39.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:39 smithi079 ceph-mon[29577]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:39.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:39 smithi079 ceph-mon[40839]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:40.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:39 smithi173 ceph-mon[33421]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:40.174 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.7 2024-09-17T09:15:40.502 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== 2024-09-17T09:15:40.503 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== == AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== ']' 2024-09-17T09:15:40.503 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:40.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:40 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2953132310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:40.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:40 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2953132310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:41.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:40 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2953132310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:41.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:41 smithi079 ceph-mon[29577]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:41.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:41 smithi079 ceph-mon[40839]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:42.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:41 smithi173 ceph-mon[33421]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:43.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:15:42 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:15:43.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:43 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:43.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:43 smithi079 ceph-mon[29577]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:43.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:43 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:15:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:15:43.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:43 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:43.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:43 smithi079 ceph-mon[40839]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:44.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:43 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:44.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:43 smithi173 ceph-mon[33421]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.7 2024-09-17T09:15:45.831 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== 2024-09-17T09:15:45.831 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== == AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== ']' 2024-09-17T09:15:45.832 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:45.832 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:45 smithi079 ceph-mon[40839]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:45.832 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:45 smithi079 ceph-mon[29577]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:46.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:45 smithi173 ceph-mon[33421]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:46 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1989255271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:46.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:46 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1989255271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:46.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:46 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1989255271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:47.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:47 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:47.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:47 smithi079 ceph-mon[40839]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:47 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:47.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:47 smithi079 ceph-mon[29577]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:48.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:47 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:48.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:47 smithi173 ceph-mon[33421]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:49.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:49 smithi079 ceph-mon[29577]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:49.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:49 smithi079 ceph-mon[40839]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:50.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:49 smithi173 ceph-mon[33421]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:50.833 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.7 2024-09-17T09:15:51.160 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== 2024-09-17T09:15:51.160 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== == AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== ']' 2024-09-17T09:15:51.160 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:51.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:51 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2898535980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:51.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:51 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2898535980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:51.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:51 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2898535980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:52.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:52 smithi173 ceph-mon[33421]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:52.611 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:52 smithi079 ceph-mon[29577]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:52.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:52 smithi079 ceph-mon[40839]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:53.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:15:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:15:53.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:53 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:53.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:53 smithi079 ceph-mon[29577]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:53.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:53 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:15:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T09:15:53.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:53 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:53.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:53 smithi079 ceph-mon[40839]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:54.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:53 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:15:54.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:53 smithi173 ceph-mon[33421]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:55.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:55 smithi079 ceph-mon[29577]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:55.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:55 smithi079 ceph-mon[40839]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:56.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:55 smithi173 ceph-mon[33421]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:15:56.162 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key osd.7 2024-09-17T09:15:56.489 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for mgr.y 2024-09-17T09:15:56.489 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQC3SOlmRQVJDhAAkrYBi9p1my3YK8KCIc+TSA== 2024-09-17T09:15:56.489 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQDLRulmM30lKBAAP9QKghq5br43hIRZwI0Y0g== == AQC3SOlmRQVJDhAAkrYBi9p1my3YK8KCIc+TSA== ']' 2024-09-17T09:15:56.489 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:15:56.489 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for mgr.y' 2024-09-17T09:15:56.489 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key mgr.y 2024-09-17T09:15:56.801 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQCvRelmdsJNKhAAMDTo8eicjAKNftsPvDX3rg== 2024-09-17T09:15:56.801 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQCvRelmdsJNKhAAMDTo8eicjAKNftsPvDX3rg== 2024-09-17T09:15:56.801 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key mgr.y 2024-09-17T09:15:57.069 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key mgr.y on host 'smithi079' 2024-09-17T09:15:57.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/426956347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:57.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:56 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/1126705380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-17T09:15:57.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/426956347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:57.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:56 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/1126705380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-17T09:15:57.081 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCvRelmdsJNKhAAMDTo8eicjAKNftsPvDX3rg== == AQCvRelmdsJNKhAAMDTo8eicjAKNftsPvDX3rg== ']' 2024-09-17T09:15:57.081 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:15:57.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/426956347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-17T09:15:57.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:56 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/1126705380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-17T09:15:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='client.25190 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.y", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:15:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: Schedule rotate-key daemon mgr.y 2024-09-17T09:15:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:58.362 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: Rotating authentication key for mgr.y 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: Reconfiguring daemon mgr.y on smithi079 2024-09-17T09:15:58.363 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[40839]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:58.364 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='client.25190 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.y", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:15:58.364 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: Schedule rotate-key daemon mgr.y 2024-09-17T09:15:58.364 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:58.364 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:58.364 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:58.364 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:58.364 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:15:58.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:58.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: Rotating authentication key for mgr.y 2024-09-17T09:15:58.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-17T09:15:58.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-17T09:15:58.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:15:58.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:15:58.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:58.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: Reconfiguring daemon mgr.y on smithi079 2024-09-17T09:15:58.365 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-mon[29577]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='client.25190 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.y", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: Schedule rotate-key daemon mgr.y 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: Rotating authentication key for mgr.y 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-17T09:15:58.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-17T09:15:58.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:15:58.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:15:58.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:15:58.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: Reconfiguring daemon mgr.y on smithi079 2024-09-17T09:15:58.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-mon[33421]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:15:58.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.712+0000 7f94dc5a5640 -1 mgr.server reply reply (13) Permission denied access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication 2024-09-17T09:15:58.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.714+0000 7f94cdd08640 -1 log_channel(cephadm) log [ERR] : Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-y/keyring', '-n', 'mgr.y', 'tell', 'mgr.y', 'rotate-key', '-i', '-']: 2024-09-17T09:15:58.706+0000 7f4f166e9640 1 Processor -- start 2024-09-17T09:15:58.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.706+0000 7f4f166e9640 1 -- start start 2024-09-17T09:15:58.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.706+0000 7f4f166e9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f1006d640 0x7f4f1006b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T09:15:58.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.706+0000 7f4f166e9640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1006bf80 con 0x7f4f1006d640 2024-09-17T09:15:58.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.706+0000 7f4f0ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f1006d640 0x7f4f1006b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T09:15:58.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.706+0000 7f4f0ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f1006d640 0x7f4f1006b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44404/0 (socket says 172.21.15.79:44404) 2024-09-17T09:15:58.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.706+0000 7f4f0ffff640 1 -- 172.21.15.79:0/855463839 learned_addr learned my addr 172.21.15.79:0/855463839 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-09-17T09:15:58.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.706+0000 7f4f0ffff640 1 -- 172.21.15.79:0/855463839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1006c0c0 con 0x7f4f1006d640 2024-09-17T09:15:58.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.707+0000 7f4f0ffff640 1 --2- 172.21.15.79:0/855463839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f1006d640 0x7f4f1006b9b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009820 tx=0x7f4f0002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe8532adbb8d6e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T09:15:58.865 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.707+0000 7f4f0effd640 1 -- 172.21.15.79:0/855463839 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00032440 con 0x7f4f1006d640 2024-09-17T09:15:58.865 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.707+0000 7f4f0effd640 1 -- 172.21.15.79:0/855463839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- auth(proto 2 2 bytes epoch 0) v1 -- 0x7f4ef8003b20 con 0x7f4f1006d640 2024-09-17T09:15:58.865 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.707+0000 7f4f0effd640 1 -- 172.21.15.79:0/855463839 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(39 keys) v1 ==== 1791+0+0 (secure 0 0 0) 0x7f4f00032d00 con 0x7f4f1006d640 2024-09-17T09:15:58.865 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.707+0000 7f4f0effd640 1 -- 172.21.15.79:0/855463839 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f000319a0 con 0x7f4f1006d640 2024-09-17T09:15:58.865 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.707+0000 7f4f0effd640 1 -- 172.21.15.79:0/855463839 <== mon.0 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7f4ef8003b20 con 0x7f4f1006d640 2024-09-17T09:15:58.866 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.707+0000 7f4f0effd640 0 cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 16428986673136858407 != 18374858748799134293 2024-09-17T09:15:58.866 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.708+0000 7f4f166e9640 1 -- 172.21.15.79:0/855463839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f1006d640 msgr2=0x7f4f1006b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T09:15:58.866 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.708+0000 7f4f166e9640 1 --2- 172.21.15.79:0/855463839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f1006d640 0x7f4f1006b9b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009820 tx=0x7f4f0002a270 comp rx=0 tx=0).stop 2024-09-17T09:15:58.866 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.708+0000 7f4f166e9640 1 -- 172.21.15.79:0/855463839 shutdown_connections 2024-09-17T09:15:58.866 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.708+0000 7f4f166e9640 1 --2- 172.21.15.79:0/855463839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f1006d640 0x7f4f1006b9b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T09:15:58.866 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.708+0000 7f4f166e9640 1 -- 172.21.15.79:0/855463839 >> 172.21.15.79:0/855463839 conn(0x7f4f1006f700 msgr2=0x7f4f10071b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T09:15:58.866 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.708+0000 7f4f166e9640 1 -- 172.21.15.79:0/855463839 shutdown_connections 2024-09-17T09:15:58.867 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.708+0000 7f4f166e9640 1 -- 172.21.15.79:0/855463839 wait complete. 2024-09-17T09:15:58.867 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.708+0000 7f4f166e9640 1 Processor -- start 2024-09-17T09:15:58.867 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.708+0000 7f4f166e9640 1 -- start start 2024-09-17T09:15:58.867 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f166e9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f1006d640 0x7f4f101a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T09:15:58.868 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f166e9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f101a2b10 0x7f4f101a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T09:15:58.868 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f166e9640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f101a2f90 0x7f4f101aa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T09:15:58.868 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f166e9640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f101aa940 con 0x7f4f101a2b10 2024-09-17T09:15:58.869 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f166e9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f101aaab0 con 0x7f4f1006d640 2024-09-17T09:15:58.870 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f166e9640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f101aac70 con 0x7f4f101a2f90 2024-09-17T09:15:58.870 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f101a2b10 0x7f4f101a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T09:15:58.870 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f101a2b10 0x7f4f101a7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44418/0 (socket says 172.21.15.79:44418) 2024-09-17T09:15:58.870 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f14c86640 1 --2- 172.21.15.79:0/2443281937 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f101a2f90 0x7f4f101aa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T09:15:58.871 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0f7fe640 1 -- 172.21.15.79:0/2443281937 learned_addr learned my addr 172.21.15.79:0/2443281937 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-09-17T09:15:58.871 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0f7fe640 1 -- 172.21.15.79:0/2443281937 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f101a2f90 msgr2=0x7f4f101aa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T09:15:58.871 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0f7fe640 1 --2- 172.21.15.79:0/2443281937 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f101a2f90 0x7f4f101aa1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T09:15:58.871 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0f7fe640 1 -- 172.21.15.79:0/2443281937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f1006d640 msgr2=0x7f4f101a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T09:15:58.872 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0ffff640 1 --2- 172.21.15.79:0/2443281937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f1006d640 0x7f4f101a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T09:15:58.872 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0f7fe640 1 --2- 172.21.15.79:0/2443281937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f1006d640 0x7f4f101a25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T09:15:58.872 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0f7fe640 1 -- 172.21.15.79:0/2443281937 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f000094d0 con 0x7f4f101a2b10 2024-09-17T09:15:58.872 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0f7fe640 1 --2- 172.21.15.79:0/2443281937 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f101a2b10 0x7f4f101a7c00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4f0400b550 tx=0x7f4f0400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T09:15:58.873 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0d7fa640 1 -- 172.21.15.79:0/2443281937 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04004230 con 0x7f4f101a2b10 2024-09-17T09:15:58.873 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0d7fa640 1 -- 172.21.15.79:0/2443281937 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- auth(proto 2 2 bytes epoch 0) v1 -- 0x7f4ef8003160 con 0x7f4f101a2b10 2024-09-17T09:15:58.873 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0d7fa640 1 -- 172.21.15.79:0/2443281937 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(39 keys) v1 ==== 1791+0+0 (secure 0 0 0) 0x7f4f0400fdb0 con 0x7f4f101a2b10 2024-09-17T09:15:58.873 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f166e9640 1 -- 172.21.15.79:0/2443281937 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f101aaf50 con 0x7f4f101a2b10 2024-09-17T09:15:58.874 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0d7fa640 1 -- 172.21.15.79:0/2443281937 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0400da80 con 0x7f4f101a2b10 2024-09-17T09:15:58.874 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f166e9640 1 -- 172.21.15.79:0/2443281937 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f101ab360 con 0x7f4f101a2b10 2024-09-17T09:15:58.874 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0d7fa640 1 -- 172.21.15.79:0/2443281937 <== mon.0 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7f4f101ab360 con 0x7f4f101a2b10 2024-09-17T09:15:58.874 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.709+0000 7f4f0d7fa640 0 cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 16428986673136858407 != 18374858748799134293 2024-09-17T09:15:58.874 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.711+0000 7f4f0d7fa640 1 -- 172.21.15.79:0/2443281937 <== mon.0 v2:172.21.15.79:3300/0 5 ==== mgrmap(e 22) v1 ==== 90785+0+0 (secure 0 0 0) 0x7f4f04004390 con 0x7f4f101a2b10 2024-09-17T09:15:58.875 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.711+0000 7f4f0d7fa640 1 --2- 172.21.15.79:0/2443281937 >> v2:172.21.15.79:6800/43857153 conn(0x7f4ef806d5c0 0x7f4ef806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T09:15:58.875 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.711+0000 7f4f0d7fa640 1 -- 172.21.15.79:0/2443281937 --> v2:172.21.15.79:6800/43857153 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ef8070150 con 0x7f4ef806d5c0 2024-09-17T09:15:58.875 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.711+0000 7f4f0d7fa640 1 -- 172.21.15.79:0/2443281937 <== mon.0 v2:172.21.15.79:3300/0 6 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f4f0408c8f0 con 0x7f4f101a2b10 2024-09-17T09:15:58.875 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.711+0000 7f4f0ffff640 1 --2- 172.21.15.79:0/2443281937 >> v2:172.21.15.79:6800/43857153 conn(0x7f4ef806d5c0 0x7f4ef806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T09:15:58.876 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.712+0000 7f4f0ffff640 1 --2- 172.21.15.79:0/2443281937 >> v2:172.21.15.79:6800/43857153 conn(0x7f4ef806d5c0 0x7f4ef806fa80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4f000097f0 tx=0x7f4f00039000 comp rx=0 tx=0).ready entity=mgr.14547 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T09:15:58.876 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.712+0000 7f4f0d7fa640 1 -- 172.21.15.79:0/2443281937 <== mgr.14547 v2:172.21.15.79:6800/43857153 1 ==== command_reply(tid 0: -13 access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication) v1 ==== 134+0+0 (secure 0 0 0) 0x7f4ef8070150 con 0x7f4ef806d5c0 2024-09-17T09:15:58.876 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: Error EACCES: problem getting command descriptions from mgr.y 2024-09-17T09:15:58.876 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.876 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.746+0000 7f9537e76640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-17T09:15:58.877 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ignoring --setuser ceph since I am not root 2024-09-17T09:15:58.877 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: ignoring --setgroup ceph since I am not root 2024-09-17T09:15:59.027 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:15:58] ENGINE Bus STOPPING 2024-09-17T09:15:59.027 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:15:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T09:15:59.027 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:15:58 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:15:58] ENGINE Bus STOPPED 2024-09-17T09:15:59.149 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.890+0000 7f74e7d22040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:15:59.149 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:58 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:58.945+0000 7f74e7d22040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:15:59.150 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:59.095+0000 7f74e7d22040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:15:59.332 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:15:59] ENGINE Bus STARTING 2024-09-17T09:15:59.332 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:15:59] ENGINE Serving on http://:::9283 2024-09-17T09:15:59.332 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:15:59] ENGINE Bus STARTED 2024-09-17T09:15:59.456 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:59.163+0000 7f74e7d22040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:15:59.456 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:59.307+0000 7f74e7d22040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:15:59.456 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:59.454+0000 7f74e7d22040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:15:59.795 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:59.795 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: Activating manager daemon x 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: mgrmap e23: x(active, starting, since 0.00970609s) 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:15:59.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: Manager daemon x is now available 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:15:59.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: Activating manager daemon x 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: mgrmap e23: x(active, starting, since 0.00970609s) 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:15:59.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:15:59.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:15:59.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: Manager daemon x is now available 2024-09-17T09:15:59.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:59.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:59.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:15:59.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:15:59.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:15:59.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:15:59.981 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: Activating manager daemon x 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.14547 172.21.15.79:0/1822690854' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: mgrmap e23: x(active, starting, since 0.00970609s) 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:15:59.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:15:59.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:15:59.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:15:59.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:15:59.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:15:59.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:15:59.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:15:59.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:15:59.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:15:59.984 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: Manager daemon x is now available 2024-09-17T09:15:59.984 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:15:59.984 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:15:59.985 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:15:59.985 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:15:59.985 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:15:59.985 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:15:59 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:16:00.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:59.792+0000 7f74e7d22040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:16:00.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:15:59 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:15:59.853+0000 7f74e7d22040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:16:00.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:00.041+0000 7f74e7d22040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:16:00.753 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:00.297+0000 7f8873f53640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:09:16:00] ENGINE Error in HTTPServer.serve 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: Traceback (most recent call last): 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: self._connections.run(self.expiration_interval) 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: self._run(expiration_interval) 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: s = self.context.wrap_socket( 2024-09-17T09:16:00.754 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:16:00.755 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: return self.sslsocket_class._create( 2024-09-17T09:16:00.755 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:16:00.755 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: self.do_handshake() 2024-09-17T09:16:00.755 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:16:00.755 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: self._sslobj.do_handshake() 2024-09-17T09:16:00.755 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:16:00.755 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:01.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-mon[33421]: mgrmap e24: x(active, since 1.01089s) 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-mon[33421]: [17/Sep/2024:09:16:00] ENGINE Bus STARTING 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-mon[33421]: [17/Sep/2024:09:16:00] ENGINE Serving on http://172.21.15.173:8765 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-mon[33421]: [17/Sep/2024:09:16:00] ENGINE Serving on https://172.21.15.173:7150 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-mon[33421]: [17/Sep/2024:09:16:00] ENGINE Bus STARTED 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:00 smithi173 ceph-mon[33421]: [17/Sep/2024:09:16:00] ENGINE Error in HTTPServer.serve 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: Traceback (most recent call last): 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._run(expiration_interval) 2024-09-17T09:16:01.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: s = self.context.wrap_socket( 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: return self.sslsocket_class._create( 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: self.do_handshake() 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._sslobj.do_handshake() 2024-09-17T09:16:01.008 INFO:journalctl@ceph.mon.b.smithi173.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:16:01.038 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:00.989+0000 7f74e7d22040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:16:01.038 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[29577]: mgrmap e24: x(active, since 1.01089s) 2024-09-17T09:16:01.038 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[29577]: [17/Sep/2024:09:16:00] ENGINE Bus STARTING 2024-09-17T09:16:01.038 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[29577]: [17/Sep/2024:09:16:00] ENGINE Serving on http://172.21.15.173:8765 2024-09-17T09:16:01.038 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[29577]: [17/Sep/2024:09:16:00] ENGINE Serving on https://172.21.15.173:7150 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[29577]: [17/Sep/2024:09:16:00] ENGINE Bus STARTED 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[29577]: [17/Sep/2024:09:16:00] ENGINE Error in HTTPServer.serve 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-17T09:16:01.039 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:16:01.040 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-17T09:16:01.040 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:16:01.040 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-17T09:16:01.040 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:16:01.040 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-17T09:16:01.040 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:16:01.040 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[40839]: mgrmap e24: x(active, since 1.01089s) 2024-09-17T09:16:01.040 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[40839]: [17/Sep/2024:09:16:00] ENGINE Bus STARTING 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[40839]: [17/Sep/2024:09:16:00] ENGINE Serving on http://172.21.15.173:8765 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[40839]: [17/Sep/2024:09:16:00] ENGINE Serving on https://172.21.15.173:7150 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[40839]: [17/Sep/2024:09:16:00] ENGINE Bus STARTED 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:00 smithi079 ceph-mon[40839]: [17/Sep/2024:09:16:00] ENGINE Error in HTTPServer.serve 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._run(expiration_interval) 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:16:01.041 INFO:journalctl@ceph.mon.c.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: s = self.context.wrap_socket( 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.sslsocket_class._create( 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.do_handshake() 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._sslobj.do_handshake() 2024-09-17T09:16:01.042 INFO:journalctl@ceph.mon.c.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:16:01.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:01 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:01.062+0000 7f74e7d22040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:16:02.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:01 smithi173 ceph-mon[33421]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:16:02.054 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:01 smithi079 ceph-mon[29577]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:16:02.054 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:01 smithi079 ceph-mon[40839]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:16:02.055 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:01 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:01.794+0000 7f74e7d22040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:16:02.055 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:01 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:01.911+0000 7f74e7d22040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:16:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key mgr.y 2024-09-17T09:16:02.360 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.052+0000 7f74e7d22040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:16:02.360 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.115+0000 7f74e7d22040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:16:02.360 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.276+0000 7f74e7d22040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:16:02.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.359+0000 7f74e7d22040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:16:02.417 INFO:teuthology.orchestra.run.smithi079.stdout:rotating key for mgr.x 2024-09-17T09:16:02.418 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQDNSOlmzpoQCRAA6l33O1BSculeJHNApZL1ew== 2024-09-17T09:16:02.418 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQCvRelmdsJNKhAAMDTo8eicjAKNftsPvDX3rg== == AQDNSOlmzpoQCRAA6l33O1BSculeJHNApZL1ew== ']' 2024-09-17T09:16:02.418 INFO:teuthology.orchestra.run.smithi079.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-17T09:16:02.418 INFO:teuthology.orchestra.run.smithi079.stderr:+ echo 'rotating key for mgr.x' 2024-09-17T09:16:02.418 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key mgr.x 2024-09-17T09:16:02.611 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.422+0000 7f74e7d22040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:16:02.612 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.590+0000 7f74e7d22040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:16:02.777 INFO:teuthology.orchestra.run.smithi079.stderr:+ K=AQAgRulmIyI9FxAA2qE2Cv5dMXeHtjU7y6oELQ== 2024-09-17T09:16:02.777 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAgRulmIyI9FxAA2qE2Cv5dMXeHtjU7y6oELQ== 2024-09-17T09:16:02.777 INFO:teuthology.orchestra.run.smithi079.stderr:+ ceph orch daemon rotate-key mgr.x 2024-09-17T09:16:02.786 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:02 smithi173 ceph-mon[33421]: mgrmap e25: x(active, since 3s) 2024-09-17T09:16:02.786 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:02 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:02.786 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:02 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:02.786 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:02 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2737307290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-17T09:16:02.947 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-mon[29577]: mgrmap e25: x(active, since 3s) 2024-09-17T09:16:02.947 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:02.947 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:02.947 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2737307290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-17T09:16:02.948 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-mon[40839]: mgrmap e25: x(active, since 3s) 2024-09-17T09:16:02.948 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:02.948 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:02.948 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2737307290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-17T09:16:02.948 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.658+0000 7f74e7d22040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:16:02.948 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.728+0000 7f74e7d22040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:16:02.948 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.819+0000 7f74e7d22040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:16:02.949 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.885+0000 7f74e7d22040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:16:02.949 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:02 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:02.946+0000 7f74e7d22040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:16:03.065 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled to rotate-key mgr.x on host 'smithi173' 2024-09-17T09:16:03.076 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAgRulmIyI9FxAA2qE2Cv5dMXeHtjU7y6oELQ== == AQAgRulmIyI9FxAA2qE2Cv5dMXeHtjU7y6oELQ== ']' 2024-09-17T09:16:03.076 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:16:03.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:16:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:16:04.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:03.723+0000 7f74e7d22040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:16:04.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:16:03] ENGINE Bus STARTING 2024-09-17T09:16:04.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: CherryPy Checker: 2024-09-17T09:16:04.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: The Application mounted at '' has an empty config. 2024-09-17T09:16:04.112 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: 2024-09-17T09:16:04.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:16:03] ENGINE Serving on http://:::9283 2024-09-17T09:16:04.113 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[29828]: [17/Sep/2024:09:16:03] ENGINE Bus STARTED 2024-09-17T09:16:04.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:16:04.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2675414364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-17T09:16:04.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:16:04.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.113 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi079 to 3248M 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: Schedule rotate-key daemon mgr.x 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.114 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: Standby manager daemon y started 2024-09-17T09:16:04.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:16:04.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:16:04.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:16:04.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:16:04.115 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:16:04.115 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2675414364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-17T09:16:04.115 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi079 to 3248M 2024-09-17T09:16:04.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: Schedule rotate-key daemon mgr.x 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: Standby manager daemon y started 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:16:04.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:16:04.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:03 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2675414364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='client.24407 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi079 to 3248M 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: Schedule rotate-key daemon mgr.x 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: Standby manager daemon y started 2024-09-17T09:16:04.258 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:16:04.259 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:16:04.259 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:16:04.259 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:03 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.79:0/4258757500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:16:05.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: Adjusting osd_memory_target on smithi173 to 3248M 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:16:05.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: mgrmap e26: x(active, since 5s), standbys: y 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.501 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.502 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: Adjusting osd_memory_target on smithi173 to 3248M 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: mgrmap e26: x(active, since 5s), standbys: y 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.503 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:05 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T09:16:05.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: Adjusting osd_memory_target on smithi173 to 3248M 2024-09-17T09:16:05.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:05.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:16:05.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-17T09:16:05.510 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T09:16:05.510 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:16:05.510 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T09:16:05.510 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: mgrmap e26: x(active, since 5s), standbys: y 2024-09-17T09:16:05.510 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:16:05.510 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.511 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.511 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.511 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:05.511 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:05 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:06.449 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:16:06.449 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:16:06.449 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:16:06.449 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:16:06.449 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: Rotating authentication key for mgr.y 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[29577]: Reconfiguring daemon mgr.y on smithi079 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:16:06.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: Rotating authentication key for mgr.y 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:06.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:06 smithi079 ceph-mon[40839]: Reconfiguring daemon mgr.y on smithi079 2024-09-17T09:16:06.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:16:06.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:16:06.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:16:06.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: Updating smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:16:06.507 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: Updating smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T09:16:06.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: Rotating authentication key for mgr.y 2024-09-17T09:16:06.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-17T09:16:06.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-17T09:16:06.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-17T09:16:06.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:06.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:06.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:16:06.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:06.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:06 smithi173 ceph-mon[33421]: Reconfiguring daemon mgr.y on smithi079 2024-09-17T09:16:07.302 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:07 smithi079 systemd[1]: Stopping Ceph mgr.y for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:07.577 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:07 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:07.577 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:07 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:07.577 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:07 smithi079 ceph-mon[29577]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T09:16:07.578 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:07 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:07.578 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:07 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:07.578 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:07 smithi079 ceph-mon[40839]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T09:16:07.578 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:07 smithi079 podman[86480]: 2024-09-17 09:16:07.301517542 +0000 UTC m=+0.131422611 container died cb364aaf426c625ee8fd9db02929fa09a36d24f80a5193f2fe2128a2fe6a289e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-17T09:16:07.844 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:07 smithi079 podman[86480]: 2024-09-17 09:16:07.666285725 +0000 UTC m=+0.496190785 container cleanup cb364aaf426c625ee8fd9db02929fa09a36d24f80a5193f2fe2128a2fe6a289e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-17T09:16:07.844 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:07 smithi079 bash[86480]: ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y 2024-09-17T09:16:08.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:07 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:08.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:07 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:08.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:07 smithi173 ceph-mon[33421]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T09:16:08.078 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key mgr.x 2024-09-17T09:16:08.108 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:07 smithi079 podman[86497]: 2024-09-17 09:16:07.843636773 +0000 UTC m=+0.540618538 container remove cb364aaf426c625ee8fd9db02929fa09a36d24f80a5193f2fe2128a2fe6a289e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:16:08.108 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:07 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:16:08.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-17T09:16:08.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 systemd[1]: Stopped Ceph mgr.y for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:16:08.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y.service: Consumed 47.340s CPU time. 2024-09-17T09:16:08.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 systemd[1]: Starting Ceph mgr.y for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:08.362 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 podman[86624]: 2024-09-17 09:16:08.245736622 +0000 UTC m=+0.021517104 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:16:08.429 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQAgRulmIyI9FxAA2qE2Cv5dMXeHtjU7y6oELQ== 2024-09-17T09:16:08.430 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAgRulmIyI9FxAA2qE2Cv5dMXeHtjU7y6oELQ== == AQAgRulmIyI9FxAA2qE2Cv5dMXeHtjU7y6oELQ== ']' 2024-09-17T09:16:08.430 INFO:teuthology.orchestra.run.smithi079.stderr:+ sleep 5 2024-09-17T09:16:08.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:08 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/2044807906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-17T09:16:08.644 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:08 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/2044807906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-17T09:16:08.644 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 podman[86624]: 2024-09-17 09:16:08.378047898 +0000 UTC m=+0.153828378 container create b1360a1cd61b9b9642a0034b5ead4c61fb93eff4cdcabcee51b646dbe90acf57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T09:16:08.644 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 podman[86624]: 2024-09-17 09:16:08.614243056 +0000 UTC m=+0.390023544 container init b1360a1cd61b9b9642a0034b5ead4c61fb93eff4cdcabcee51b646dbe90acf57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-17T09:16:08.644 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 podman[86624]: 2024-09-17 09:16:08.619050559 +0000 UTC m=+0.394831047 container start b1360a1cd61b9b9642a0034b5ead4c61fb93eff4cdcabcee51b646dbe90acf57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-17T09:16:08.644 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 ceph-mgr[86668]: -- 172.21.15.79:0/2880595263 <== mon.2 v2:172.21.15.79:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5590486a0d00 con 0x559048674800 2024-09-17T09:16:08.953 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 ceph-mgr[86668]: cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 9050974532026146351 != 18374858748799134293 2024-09-17T09:16:08.954 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 bash[86624]: b1360a1cd61b9b9642a0034b5ead4c61fb93eff4cdcabcee51b646dbe90acf57 2024-09-17T09:16:08.954 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 systemd[1]: Started Ceph mgr.y for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:16:08.954 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:08.740+0000 7f45b973c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:16:08.954 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:08.803+0000 7f45b973c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:16:08.954 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:08 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:08.952+0000 7f45b973c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:16:09.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:08 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/2044807906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-17T09:16:09.300 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:09.018+0000 7f45b973c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:16:09.301 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:09.161+0000 7f45b973c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:16:09.607 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:09.299+0000 7f45b973c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:16:09.607 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:09.606+0000 7f45b973c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:16:09.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:09.665+0000 7f45b973c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:16:09.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:09.849+0000 7f45b973c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:16:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: Rotating authentication key for mgr.x 2024-09-17T09:16:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-17T09:16:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-17T09:16:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T09:16:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]': finished 2024-09-17T09:16:09.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[29577]: Reconfiguring daemon mgr.x on smithi173 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: Rotating authentication key for mgr.x 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]': finished 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:09.863 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:09.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:16:09.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:09.864 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:09 smithi079 ceph-mon[40839]: Reconfiguring daemon mgr.x on smithi173 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: Rotating authentication key for mgr.x 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]': finished 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:16:10.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:10.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:09 smithi173 ceph-mon[33421]: Reconfiguring daemon mgr.x on smithi173 2024-09-17T09:16:10.476 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.322+0000 7f88aefc9640 -1 mgr.server reply reply (13) Permission denied access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication 2024-09-17T09:16:10.476 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.325+0000 7f88ac7c4640 -1 log_channel(cephadm) log [ERR] : Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-x/keyring', '-n', 'mgr.x', 'tell', 'mgr.x', 'rotate-key', '-i', '-']: 2024-09-17T09:16:10.318+0000 7fa7beffd640 1 -- 172.21.15.173:0/3453153436 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7fa7ac003750 con 0x7fa7c0109890 2024-09-17T09:16:10.476 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.318+0000 7fa7beffd640 0 cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 14882593707904250136 != 18374858748799134293 2024-09-17T09:16:10.476 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.318+0000 7fa7c65cb640 1 -- 172.21.15.173:0/3453153436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7c0109890 msgr2=0x7fa7c0109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T09:16:10.476 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.318+0000 7fa7c65cb640 1 --2- 172.21.15.173:0/3453153436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7c0109890 0x7fa7c0109d10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009900 tx=0x7fa7b002a410 comp rx=0 tx=0).stop 2024-09-17T09:16:10.476 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 -- 172.21.15.173:0/3453153436 shutdown_connections 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 --2- 172.21.15.173:0/3453153436 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa7c010a250 0x7fa7c0110b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 --2- 172.21.15.173:0/3453153436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7c0109890 0x7fa7c0109d10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 --2- 172.21.15.173:0/3453153436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7c0108850 0x7fa7c0108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 -- 172.21.15.173:0/3453153436 >> 172.21.15.173:0/3453153436 conn(0x7fa7c006f0c0 msgr2=0x7fa7c0071500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 -- 172.21.15.173:0/3453153436 shutdown_connections 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 -- 172.21.15.173:0/3453153436 wait complete. 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 Processor -- start 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 -- start start 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7c0108850 0x7fa7c006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T09:16:10.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa7c0109890 0x7fa7c00699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7c010a250 0x7fa7c0069ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c006a680 con 0x7fa7c0108850 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c006a7f0 con 0x7fa7c010a250 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c65cb640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c01a7f90 con 0x7fa7c0109890 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c4b68640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7c010a250 0x7fa7c0069ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c4b68640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7c010a250 0x7fa7c0069ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:36012/0 (socket says 172.21.15.173:36012) 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7c4b68640 1 -- 172.21.15.173:0/2188697172 learned_addr learned my addr 172.21.15.173:0/2188697172 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.319+0000 7fa7bffff640 1 --2- 172.21.15.173:0/2188697172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7c0108850 0x7fa7c006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7bf7fe640 1 --2- 172.21.15.173:0/2188697172 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa7c0109890 0x7fa7c00699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7c4b68640 1 -- 172.21.15.173:0/2188697172 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa7c0109890 msgr2=0x7fa7c00699b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T09:16:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7c4b68640 1 --2- 172.21.15.173:0/2188697172 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa7c0109890 0x7fa7c00699b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T09:16:10.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7c4b68640 1 -- 172.21.15.173:0/2188697172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7c0108850 msgr2=0x7fa7c006b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T09:16:10.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7c4b68640 1 --2- 172.21.15.173:0/2188697172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7c0108850 0x7fa7c006b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T09:16:10.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7c4b68640 1 -- 172.21.15.173:0/2188697172 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0009560 con 0x7fa7c010a250 2024-09-17T09:16:10.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7c4b68640 1 --2- 172.21.15.173:0/2188697172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7c010a250 0x7fa7c0069ef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800ea20 tx=0x7fa7b800eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T09:16:10.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7bd7fa640 1 -- 172.21.15.173:0/2188697172 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b800cde0 con 0x7fa7c010a250 2024-09-17T09:16:10.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7bd7fa640 1 -- 172.21.15.173:0/2188697172 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- auth(proto 2 2 bytes epoch 0) v1 -- 0x7fa7ac003310 con 0x7fa7c010a250 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7bd7fa640 1 -- 172.21.15.173:0/2188697172 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(39 keys) v1 ==== 1791+0+0 (secure 0 0 0) 0x7fa7b8005780 con 0x7fa7c010a250 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7c65cb640 1 -- 172.21.15.173:0/2188697172 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c01a8240 con 0x7fa7c010a250 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7bd7fa640 1 -- 172.21.15.173:0/2188697172 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8010430 con 0x7fa7c010a250 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7c65cb640 1 -- 172.21.15.173:0/2188697172 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c01a8760 con 0x7fa7c010a250 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7bd7fa640 1 -- 172.21.15.173:0/2188697172 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7fa7c01a8760 con 0x7fa7c010a250 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.320+0000 7fa7bd7fa640 0 cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 14882593707904250136 != 18374858748799134293 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.322+0000 7fa7bd7fa640 1 -- 172.21.15.173:0/2188697172 <== mon.1 v2:172.21.15.173:3300/0 5 ==== mgrmap(e 26) v1 ==== 90787+0+0 (secure 0 0 0) 0x7fa7b8010860 con 0x7fa7c010a250 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.322+0000 7fa7bd7fa640 1 --2- 172.21.15.173:0/2188697172 >> v2:172.21.15.173:6816/3039282380 conn(0x7fa7ac06d2e0 0x7fa7ac06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.322+0000 7fa7bd7fa640 1 -- 172.21.15.173:0/2188697172 --> v2:172.21.15.173:6816/3039282380 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7ac06fe70 con 0x7fa7ac06d2e0 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.322+0000 7fa7bd7fa640 1 -- 172.21.15.173:0/2188697172 <== mon.1 v2:172.21.15.173:3300/0 6 ==== osd_map(72..72 src has 1..72) v4 ==== 6474+0+0 (secure 0 0 0) 0x7fa7b808d290 con 0x7fa7c010a250 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.322+0000 7fa7bffff640 1 --2- 172.21.15.173:0/2188697172 >> v2:172.21.15.173:6816/3039282380 conn(0x7fa7ac06d2e0 0x7fa7ac06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T09:16:10.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.322+0000 7fa7bffff640 1 --2- 172.21.15.173:0/2188697172 >> v2:172.21.15.173:6816/3039282380 conn(0x7fa7ac06d2e0 0x7fa7ac06f7a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa7c006b180 tx=0x7fa7a800b390 comp rx=0 tx=0).ready entity=mgr.24460 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T09:16:10.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.323+0000 7fa7bd7fa640 1 -- 172.21.15.173:0/2188697172 <== mgr.24460 v2:172.21.15.173:6816/3039282380 1 ==== command_reply(tid 0: -13 access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication) v1 ==== 134+0+0 (secure 0 0 0) 0x7fa7ac06fe70 con 0x7fa7ac06d2e0 2024-09-17T09:16:10.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: Error EACCES: problem getting command descriptions from mgr.x 2024-09-17T09:16:10.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.337+0000 7f8916b72640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-17T09:16:10.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: ignoring --setuser ceph since I am not root 2024-09-17T09:16:10.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: ignoring --setgroup ceph since I am not root 2024-09-17T09:16:10.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-mgr[35337]: -- 172.21.15.173:0/776782722 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e942afad00 con 0x55e942acf000 2024-09-17T09:16:10.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-mgr[35337]: cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 14882593707904250136 != 18374858748799134293 2024-09-17T09:16:10.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.474+0000 7f7c04e4e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:16:10.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.530+0000 7f7c04e4e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:16:10.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.677+0000 7f7c04e4e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:16:11.006 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.742+0000 7f7c04e4e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:16:11.006 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:10.888+0000 7f7c04e4e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:16:11.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:10 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:10.728+0000 7f45b973c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:16:11.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:10 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:10.793+0000 7f45b973c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:16:11.337 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:11.028+0000 7f7c04e4e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:16:11.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:11.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:11.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-mon[33421]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:16:11.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:16:11.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-mon[33421]: Activating manager daemon y 2024-09-17T09:16:11.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-mon[33421]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T09:16:11.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-mon[33421]: from='mgr.24460 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T09:16:11.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-mon[33421]: mgrmap e27: y(active, starting, since 0.00677994s) 2024-09-17T09:16:11.511 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:11.511 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:11.511 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[40839]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:16:11.511 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:16:11.512 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[40839]: Activating manager daemon y 2024-09-17T09:16:11.512 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[40839]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T09:16:11.512 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[40839]: from='mgr.24460 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T09:16:11.512 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[40839]: mgrmap e27: y(active, starting, since 0.00677994s) 2024-09-17T09:16:11.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:11.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' 2024-09-17T09:16:11.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[29577]: from='mgr.24460 172.21.15.173:0/2216114361' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:16:11.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:16:11.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[29577]: Activating manager daemon y 2024-09-17T09:16:11.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[29577]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T09:16:11.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[29577]: from='mgr.24460 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T09:16:11.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-mon[29577]: mgrmap e27: y(active, starting, since 0.00677994s) 2024-09-17T09:16:11.756 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:11.335+0000 7f7c04e4e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:16:11.757 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:11.397+0000 7f7c04e4e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:16:11.757 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:11.576+0000 7f7c04e4e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:16:11.766 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:11.510+0000 7f45b973c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:16:11.766 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:11.625+0000 7f45b973c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:16:11.766 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:11.765+0000 7f45b973c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:16:12.037 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:11.826+0000 7f45b973c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:16:12.037 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:11.969+0000 7f45b973c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:16:12.037 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:12.035+0000 7f45b973c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:16:12.316 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:12.094+0000 7f45b973c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:16:12.316 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:12.248+0000 7f45b973c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:16:12.316 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:12.315+0000 7f45b973c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:16:12.586 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:12.388+0000 7f45b973c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:16:12.587 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:12.469+0000 7f45b973c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:16:12.587 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:12.527+0000 7f45b973c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:16:12.587 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:12 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:12.585+0000 7f45b973c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:16:12.756 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:12.437+0000 7f7c04e4e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:16:12.756 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:12.502+0000 7f7c04e4e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:16:13.432 INFO:teuthology.orchestra.run.smithi079.stderr:++ ceph auth get-key mgr.x 2024-09-17T09:16:13.462 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:13.205+0000 7f7c04e4e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:16:13.463 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:13.318+0000 7f7c04e4e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:16:13.463 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: Active manager daemon y restarted 2024-09-17T09:16:13.463 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: Activating manager daemon y 2024-09-17T09:16:13.463 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T09:16:13.463 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: mgrmap e28: y(active, starting, since 0.00548199s) 2024-09-17T09:16:13.463 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.? 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:16:13.463 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:16:13.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:16:13.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:16:13.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:16:13.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:16:13.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:16:13.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:16:13.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: Active manager daemon y restarted 2024-09-17T09:16:13.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: Activating manager daemon y 2024-09-17T09:16:13.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T09:16:13.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: mgrmap e28: y(active, starting, since 0.00548199s) 2024-09-17T09:16:13.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.? 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:16:13.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:16:13.612 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:16:13.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:16:13.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:16:13.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:16:13.613 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:16:13.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:16:13.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:16:13.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:16:13.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:16:13.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:16:13.615 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:16:13.615 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:16:13.615 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:16:13.615 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:16:13.615 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:16:13.615 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:16:13.616 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:16:13.616 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:16:13.617 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:13.359+0000 7f45b973c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:16:13.617 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: [17/Sep/2024:09:16:13] ENGINE Bus STARTING 2024-09-17T09:16:13.618 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: CherryPy Checker: 2024-09-17T09:16:13.618 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: The Application mounted at '' has an empty config. 2024-09-17T09:16:13.618 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:13.618 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: [17/Sep/2024:09:16:13] ENGINE Serving on http://:::9283 2024-09-17T09:16:13.618 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: [17/Sep/2024:09:16:13] ENGINE Bus STARTED 2024-09-17T09:16:13.619 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: [17/Sep/2024:09:16:13] ENGINE Bus STOPPING 2024-09-17T09:16:13.619 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: ::ffff:172.21.15.173 - - [17/Sep/2024:09:16:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-17T09:16:13.619 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: [17/Sep/2024:09:16:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T09:16:13.619 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: [17/Sep/2024:09:16:13] ENGINE Bus STOPPED 2024-09-17T09:16:13.619 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: [17/Sep/2024:09:16:13] ENGINE Bus STARTING 2024-09-17T09:16:13.620 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: Active manager daemon y restarted 2024-09-17T09:16:13.620 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: Activating manager daemon y 2024-09-17T09:16:13.620 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T09:16:13.620 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: mgrmap e28: y(active, starting, since 0.00548199s) 2024-09-17T09:16:13.620 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.? 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:16:13.621 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:16:13.621 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:16:13.621 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:16:13.621 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:16:13.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:16:13.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:16:13.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:16:13.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:16:13.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:16:13.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:16:13.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:16:13.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:16:13.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:16:13.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:16:13.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:16:13.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:16:13.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:16:13.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:16:13.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:16:13.738 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:13.461+0000 7f7c04e4e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:16:13.738 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:13.526+0000 7f7c04e4e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:16:13.738 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:13.674+0000 7f7c04e4e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:16:13.780 INFO:teuthology.orchestra.run.smithi079.stderr:+ NK=AQDYSOlmJ6T0LBAAzsGkBWnbMAs1lw+Xbns6HQ== 2024-09-17T09:16:13.780 INFO:teuthology.orchestra.run.smithi079.stderr:+ '[' AQAgRulmIyI9FxAA2qE2Cv5dMXeHtjU7y6oELQ== == AQDYSOlmJ6T0LBAAzsGkBWnbMAs1lw+Xbns6HQ== ']' 2024-09-17T09:16:14.006 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:13.742+0000 7f7c04e4e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:16:14.006 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:13.801+0000 7f7c04e4e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:16:14.007 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:13.962+0000 7f7c04e4e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:16:14.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: [17/Sep/2024:09:16:13] ENGINE Serving on http://:::9283 2024-09-17T09:16:14.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:13 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: [17/Sep/2024:09:16:13] ENGINE Bus STARTED 2024-09-17T09:16:14.332 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:14.030+0000 7f7c04e4e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:16:14.333 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:14.100+0000 7f7c04e4e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:16:14.333 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:14.192+0000 7f7c04e4e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:16:14.333 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:14.264+0000 7f7c04e4e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:16:14.333 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:14.331+0000 7f7c04e4e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:16:14.414 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-17T09:16:14.423 INFO:tasks.cephadm:Teardown begin 2024-09-17T09:16:14.424 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:16:14.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:16:14.480 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-17T09:16:14.480 DEBUG:teuthology.orchestra.run.smithi079:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T09:16:14.510 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T09:16:14.535 INFO:tasks.cephadm:Stopping all daemons... 2024-09-17T09:16:14.535 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-17T09:16:14.535 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.a 2024-09-17T09:16:14.696 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug there is no tcmu-runner data available 2024-09-17T09:16:14.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-mon[33421]: Manager daemon y is now available 2024-09-17T09:16:14.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:16:14.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:16:14.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:16:14.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-mon[33421]: from='mgr.15408 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:16:14.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-mon[33421]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:16:14.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-mon[33421]: from='mgr.15408 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:16:14.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:14 smithi173 ceph-mon[33421]: from='client.? 172.21.15.79:0/3586826473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-17T09:16:14.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[40839]: Manager daemon y is now available 2024-09-17T09:16:14.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[40839]: from='mgr.15408 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[40839]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[40839]: from='mgr.15408 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[40839]: from='client.? 172.21.15.79:0/3586826473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[29577]: Manager daemon y is now available 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:16:14.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[29577]: from='mgr.15408 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:16:14.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[29577]: from='mgr.15408 172.21.15.79:0/1600600330' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:16:14.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[29577]: from='mgr.15408 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:16:14.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:14 smithi079 ceph-mon[29577]: from='client.? 172.21.15.79:0/3586826473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-17T09:16:14.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:14 smithi079 systemd[1]: Stopping Ceph mon.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:15.439 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:15.170+0000 7f7c04e4e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:16:15.440 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:16:15] ENGINE Bus STARTING 2024-09-17T09:16:15.440 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: CherryPy Checker: 2024-09-17T09:16:15.440 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: The Application mounted at '' has an empty config. 2024-09-17T09:16:15.440 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: 2024-09-17T09:16:15.440 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:16:15] ENGINE Serving on http://:::9283 2024-09-17T09:16:15.440 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x[35314]: [17/Sep/2024:09:16:15] ENGINE Bus STARTED 2024-09-17T09:16:15.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a[29554]: 2024-09-17T09:16:15.386+0000 7fef4498b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:16:15.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a[29554]: 2024-09-17T09:16:15.386+0000 7fef4498b640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-17T09:16:15.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:15 smithi079 podman[87180]: 2024-09-17 09:16:15.569940684 +0000 UTC m=+0.292953856 container died add4676afaeed12f8a9808966d369f6c420791ee4ac18b758b05b11527f97597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:16:15.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:15.574+0000 7f4511b2f640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:09:16:15] ENGINE Error in HTTPServer.serve 2024-09-17T09:16:15.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: Traceback (most recent call last): 2024-09-17T09:16:15.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:16:15.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: self._connections.run(self.expiration_interval) 2024-09-17T09:16:15.862 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: self._run(expiration_interval) 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: s = self.context.wrap_socket( 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: return self.sslsocket_class._create( 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: self.do_handshake() 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: self._sslobj.do_handshake() 2024-09-17T09:16:15.863 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:16:15.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:15 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y[86645]: 2024-09-17T09:16:16.202 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:15 smithi079 podman[87180]: 2024-09-17 09:16:15.959608329 +0000 UTC m=+0.682621498 container cleanup add4676afaeed12f8a9808966d369f6c420791ee4ac18b758b05b11527f97597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-17T09:16:16.202 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 17 09:16:15 smithi079 bash[87180]: ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-a 2024-09-17T09:16:16.350 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.a.service' 2024-09-17T09:16:16.417 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:16:16.417 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-17T09:16:16.418 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-17T09:16:16.418 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.c 2024-09-17T09:16:16.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:16 smithi079 systemd[1]: Stopping Ceph mon.c for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:17.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:17 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-c[40816]: 2024-09-17T09:16:17.328+0000 7f0ef9984640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:16:17.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:17 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-c[40816]: 2024-09-17T09:16:17.328+0000 7f0ef9984640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-17T09:16:17.612 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:17 smithi079 podman[87598]: 2024-09-17 09:16:17.4339649 +0000 UTC m=+0.339099726 container died 7d28a3dedfcafb44ca5a3f5f45dcd718461898ef99b229b1774a80197a6c34ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-17T09:16:17.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:17 smithi079 podman[87598]: 2024-09-17 09:16:17.736244881 +0000 UTC m=+0.641379701 container cleanup 7d28a3dedfcafb44ca5a3f5f45dcd718461898ef99b229b1774a80197a6c34ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-17T09:16:17.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:17 smithi079 bash[87598]: ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-c 2024-09-17T09:16:18.273 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:17 smithi079 podman[87649]: 2024-09-17 09:16:17.929534941 +0000 UTC m=+0.545028745 container remove 7d28a3dedfcafb44ca5a3f5f45dcd718461898ef99b229b1774a80197a6c34ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:16:18.483 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.c.service' 2024-09-17T09:16:18.546 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:18 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-17T09:16:18.546 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:18 smithi079 systemd[1]: Stopped Ceph mon.c for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:16:18.546 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 17 09:16:18 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.c.service: Consumed 6.805s CPU time. 2024-09-17T09:16:18.561 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:16:18.561 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-17T09:16:18.561 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-17T09:16:18.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.b 2024-09-17T09:16:18.884 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:18 smithi173 systemd[1]: Stopping Ceph mon.b for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:19.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:18 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-b[33398]: 2024-09-17T09:16:18.881+0000 7fab4658a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:16:19.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:18 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-b[33398]: 2024-09-17T09:16:18.881+0000 7fab4658a640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-17T09:16:19.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:19 smithi173 podman[73591]: 2024-09-17 09:16:19.015081476 +0000 UTC m=+0.263845597 container died 6f8b6ca97f1577b52125cf31d38f11697715d2db67b175122482d05def08d7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:16:19.707 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:19 smithi173 podman[73591]: 2024-09-17 09:16:19.371528426 +0000 UTC m=+0.620292540 container cleanup 6f8b6ca97f1577b52125cf31d38f11697715d2db67b175122482d05def08d7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-09-17T09:16:19.707 INFO:journalctl@ceph.mon.b.smithi173.stdout:Sep 17 09:16:19 smithi173 bash[73591]: ceph-721af936-74d3-11ef-bceb-c7b262605968-mon-b 2024-09-17T09:16:19.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mon.b.service' 2024-09-17T09:16:19.919 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:16:19.919 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-17T09:16:19.920 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-17T09:16:19.920 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y 2024-09-17T09:16:20.245 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:19 smithi079 systemd[1]: Stopping Ceph mgr.y for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:20.544 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:20 smithi079 podman[87764]: 2024-09-17 09:16:20.250703507 +0000 UTC m=+0.136907174 container died b1360a1cd61b9b9642a0034b5ead4c61fb93eff4cdcabcee51b646dbe90acf57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-17T09:16:20.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:20 smithi079 podman[87764]: 2024-09-17 09:16:20.643056497 +0000 UTC m=+0.529260188 container cleanup b1360a1cd61b9b9642a0034b5ead4c61fb93eff4cdcabcee51b646dbe90acf57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:16:20.861 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 17 09:16:20 smithi079 bash[87764]: ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-y 2024-09-17T09:16:20.996 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-17T09:16:21.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:16:21.031 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-17T09:16:21.032 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-17T09:16:21.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.x 2024-09-17T09:16:21.378 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:21 smithi173 systemd[1]: Stopping Ceph mgr.x for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:21.644 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:21 smithi173 podman[73727]: 2024-09-17 09:16:21.377309844 +0000 UTC m=+0.160512249 container died fa80598b963d6dd2760903eed4d79c8d838e3e6d735c90cda278dc588950226c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-17T09:16:22.006 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:21 smithi173 podman[73727]: 2024-09-17 09:16:21.751583347 +0000 UTC m=+0.534785740 container cleanup fa80598b963d6dd2760903eed4d79c8d838e3e6d735c90cda278dc588950226c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b) 2024-09-17T09:16:22.007 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Sep 17 09:16:21 smithi173 bash[73727]: ceph-721af936-74d3-11ef-bceb-c7b262605968-mgr-x 2024-09-17T09:16:22.111 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:21 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:21.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:16:22.111 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:21 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:21.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:16:22.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-17T09:16:22.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:16:22.206 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-17T09:16:22.206 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-17T09:16:22.207 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.0 2024-09-17T09:16:22.532 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:22 smithi079 systemd[1]: Stopping Ceph osd.0 for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:22.861 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:22 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0[45195]: 2024-09-17T09:16:22.531+0000 7fa50c795640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:16:22.862 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:22 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0[45195]: 2024-09-17T09:16:22.531+0000 7fa50c795640 -1 osd.0 74 *** Got signal Terminated *** 2024-09-17T09:16:22.862 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:22 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0[45195]: 2024-09-17T09:16:22.531+0000 7fa50c795640 -1 osd.0 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:16:28.232 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:27 smithi079 podman[87895]: 2024-09-17 09:16:27.956726544 +0000 UTC m=+5.542046210 container died e3ab4372c0a952181e54c9605059652997be28a67ba0e27aa8a07dac693261a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b) 2024-09-17T09:16:28.589 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:28 smithi079 podman[87895]: 2024-09-17 09:16:28.337911678 +0000 UTC m=+5.923231342 container cleanup e3ab4372c0a952181e54c9605059652997be28a67ba0e27aa8a07dac693261a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T09:16:28.589 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:28 smithi079 bash[87895]: ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0 2024-09-17T09:16:28.861 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:28 smithi079 podman[87915]: 2024-09-17 09:16:28.589036904 +0000 UTC m=+0.624093781 container remove e3ab4372c0a952181e54c9605059652997be28a67ba0e27aa8a07dac693261a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0) 2024-09-17T09:16:29.218 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:28 smithi079 podman[88009]: 2024-09-17 09:16:28.948917346 +0000 UTC m=+0.113171487 container create dfaf0cd34fa929fc189471e2f8d85cf5fef5226fc9137767b746dd045319f260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:16:29.218 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:28 smithi079 podman[88009]: 2024-09-17 09:16:28.852361755 +0000 UTC m=+0.016615851 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:16:29.219 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:29 smithi079 podman[88009]: 2024-09-17 09:16:29.142816532 +0000 UTC m=+0.307070618 container init dfaf0cd34fa929fc189471e2f8d85cf5fef5226fc9137767b746dd045319f260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:16:29.219 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:29 smithi079 podman[88009]: 2024-09-17 09:16:29.147232573 +0000 UTC m=+0.311486660 container start dfaf0cd34fa929fc189471e2f8d85cf5fef5226fc9137767b746dd045319f260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T09:16:29.219 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:29 smithi079 podman[88009]: 2024-09-17 09:16:29.217640843 +0000 UTC m=+0.381894922 container attach dfaf0cd34fa929fc189471e2f8d85cf5fef5226fc9137767b746dd045319f260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-17T09:16:29.612 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 17 09:16:29 smithi079 podman[88009]: 2024-09-17 09:16:29.337698692 +0000 UTC m=+0.501952778 container died dfaf0cd34fa929fc189471e2f8d85cf5fef5226fc9137767b746dd045319f260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:16:29.912 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.0.service' 2024-09-17T09:16:29.950 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:16:29.951 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-17T09:16:29.951 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-17T09:16:29.951 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.1 2024-09-17T09:16:30.268 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:30 smithi079 systemd[1]: Stopping Ceph osd.1 for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:30.611 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1[51422]: 2024-09-17T09:16:30.266+0000 7f2fcee91640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:16:30.611 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1[51422]: 2024-09-17T09:16:30.266+0000 7f2fcee91640 -1 osd.1 74 *** Got signal Terminated *** 2024-09-17T09:16:30.612 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:30 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1[51422]: 2024-09-17T09:16:30.266+0000 7f2fcee91640 -1 osd.1 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:16:32.111 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:31.648Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:16:32.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:31.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:16:32.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:31.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:16:36.032 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:35 smithi079 podman[88138]: 2024-09-17 09:16:35.745665503 +0000 UTC m=+5.573047026 container died b979467be224dab5fa202fb48e667180c81319392652bc8fc794fe3d04a55bb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:16:36.361 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:36 smithi079 podman[88138]: 2024-09-17 09:16:36.122965987 +0000 UTC m=+5.950347499 container cleanup b979467be224dab5fa202fb48e667180c81319392652bc8fc794fe3d04a55bb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:16:36.361 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:36 smithi079 bash[88138]: ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1 2024-09-17T09:16:36.675 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:36 smithi079 podman[88162]: 2024-09-17 09:16:36.365765519 +0000 UTC m=+0.617560440 container remove b979467be224dab5fa202fb48e667180c81319392652bc8fc794fe3d04a55bb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:16:36.961 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:36 smithi079 podman[88255]: 2024-09-17 09:16:36.675015042 +0000 UTC m=+0.113098537 container create 9ee68578c90eaac3e199a2088158391b7803e27d9bf2ce342df9e97992186407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-17T09:16:36.961 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:36 smithi079 podman[88255]: 2024-09-17 09:16:36.577916137 +0000 UTC m=+0.015999642 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:16:36.961 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:36 smithi079 podman[88255]: 2024-09-17 09:16:36.869214415 +0000 UTC m=+0.307297918 container init 9ee68578c90eaac3e199a2088158391b7803e27d9bf2ce342df9e97992186407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-17T09:16:36.961 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:36 smithi079 podman[88255]: 2024-09-17 09:16:36.874364932 +0000 UTC m=+0.312448435 container start 9ee68578c90eaac3e199a2088158391b7803e27d9bf2ce342df9e97992186407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=) 2024-09-17T09:16:36.961 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:36 smithi079 podman[88255]: 2024-09-17 09:16:36.960346125 +0000 UTC m=+0.398429621 container attach 9ee68578c90eaac3e199a2088158391b7803e27d9bf2ce342df9e97992186407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-17T09:16:37.347 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 17 09:16:37 smithi079 podman[88255]: 2024-09-17 09:16:37.059963835 +0000 UTC m=+0.498047331 container died 9ee68578c90eaac3e199a2088158391b7803e27d9bf2ce342df9e97992186407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-17T09:16:37.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.1.service' 2024-09-17T09:16:37.652 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:16:37.652 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-17T09:16:37.652 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-17T09:16:37.652 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.2 2024-09-17T09:16:37.861 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:37 smithi079 systemd[1]: Stopping Ceph osd.2 for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:38.361 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:37 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2[57821]: 2024-09-17T09:16:37.959+0000 7f865aa74640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:16:38.362 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:37 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2[57821]: 2024-09-17T09:16:37.959+0000 7f865aa74640 -1 osd.2 74 *** Got signal Terminated *** 2024-09-17T09:16:38.362 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:37 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2[57821]: 2024-09-17T09:16:37.959+0000 7f865aa74640 -1 osd.2 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:16:42.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:41 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:41.649Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:16:42.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:41 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:41.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:16:42.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:41 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:41.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:16:43.608 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:43 smithi079 podman[88384]: 2024-09-17 09:16:43.29616857 +0000 UTC m=+5.422410098 container died 7cdaeb8f15289db6c603c6dd39cf7096b2b533ddbe0cdee177ea6980d626bb3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-17T09:16:43.608 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:43 smithi079 podman[88384]: 2024-09-17 09:16:43.607748028 +0000 UTC m=+5.733989545 container cleanup 7cdaeb8f15289db6c603c6dd39cf7096b2b533ddbe0cdee177ea6980d626bb3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T09:16:43.861 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:43 smithi079 bash[88384]: ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2 2024-09-17T09:16:43.862 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:43 smithi079 podman[88415]: 2024-09-17 09:16:43.809076311 +0000 UTC m=+0.510394334 container remove 7cdaeb8f15289db6c603c6dd39cf7096b2b533ddbe0cdee177ea6980d626bb3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T09:16:44.504 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:44 smithi079 podman[88508]: 2024-09-17 09:16:44.13009661 +0000 UTC m=+0.016059935 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:16:44.504 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:44 smithi079 podman[88508]: 2024-09-17 09:16:44.234682671 +0000 UTC m=+0.120645996 container create dd96aef79fe5452aa229c46d573540f98dd3cf4392acb00743916938444ffe38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T09:16:44.504 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:44 smithi079 podman[88508]: 2024-09-17 09:16:44.420351868 +0000 UTC m=+0.306315193 container init dd96aef79fe5452aa229c46d573540f98dd3cf4392acb00743916938444ffe38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T09:16:44.504 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:44 smithi079 podman[88508]: 2024-09-17 09:16:44.425448627 +0000 UTC m=+0.311411956 container start dd96aef79fe5452aa229c46d573540f98dd3cf4392acb00743916938444ffe38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-17T09:16:44.504 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:44 smithi079 podman[88508]: 2024-09-17 09:16:44.503375155 +0000 UTC m=+0.389338499 container attach dd96aef79fe5452aa229c46d573540f98dd3cf4392acb00743916938444ffe38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:16:44.861 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:44 smithi079 podman[88508]: 2024-09-17 09:16:44.6172463 +0000 UTC m=+0.503209626 container died dd96aef79fe5452aa229c46d573540f98dd3cf4392acb00743916938444ffe38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-17T09:16:45.182 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.2.service' 2024-09-17T09:16:45.207 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:45 smithi079 podman[88551]: 2024-09-17 09:16:45.165091494 +0000 UTC m=+0.537487847 container remove dd96aef79fe5452aa229c46d573540f98dd3cf4392acb00743916938444ffe38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:16:45.207 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:45 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-17T09:16:45.207 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:45 smithi079 systemd[1]: Stopped Ceph osd.2 for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:16:45.207 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 17 09:16:45 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.2.service: Consumed 5.316s CPU time. 2024-09-17T09:16:45.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:16:45.223 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-17T09:16:45.223 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-17T09:16:45.223 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.3 2024-09-17T09:16:45.545 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:45 smithi079 systemd[1]: Stopping Ceph osd.3 for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:45.861 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:45 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3[64427]: 2024-09-17T09:16:45.544+0000 7fa9e5550640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:16:45.861 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:45 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3[64427]: 2024-09-17T09:16:45.544+0000 7fa9e5550640 -1 osd.3 74 *** Got signal Terminated *** 2024-09-17T09:16:45.862 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:45 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3[64427]: 2024-09-17T09:16:45.544+0000 7fa9e5550640 -1 osd.3 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:16:49.507 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:49 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:49.080+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:49.508 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:49 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:49.206+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:50.244 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:49 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:49.924+0000 7f73b22a3640 -1 osd.4 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.595531+0000 front 2024-09-17T09:16:23.595523+0000 (oldest deadline 2024-09-17T09:16:49.495412+0000) 2024-09-17T09:16:50.244 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:50 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:50.100+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:50.245 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:49 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:49.945+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:50.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:50 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:50.242+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:51.160 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:50 smithi079 podman[88635]: 2024-09-17 09:16:50.892127563 +0000 UTC m=+5.443963451 container died 7f82f0f5d2254e3b70ef995f9a934611be6821088b726be436609399b724c677 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T09:16:51.235 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:50 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:50.888+0000 7f73b22a3640 -1 osd.4 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.595531+0000 front 2024-09-17T09:16:23.595523+0000 (oldest deadline 2024-09-17T09:16:49.495412+0000) 2024-09-17T09:16:51.235 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:51.089+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:51.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:50 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:50.975+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:51.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:51.234+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:51.518 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:51 smithi079 podman[88635]: 2024-09-17 09:16:51.266843233 +0000 UTC m=+5.818679120 container cleanup 7f82f0f5d2254e3b70ef995f9a934611be6821088b726be436609399b724c677 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:16:51.518 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:51 smithi079 bash[88635]: ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3 2024-09-17T09:16:51.802 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:51 smithi079 podman[88655]: 2024-09-17 09:16:51.517961895 +0000 UTC m=+0.620070331 container remove 7f82f0f5d2254e3b70ef995f9a934611be6821088b726be436609399b724c677 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T09:16:51.802 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:51 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:51.650Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:16:51.802 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:51 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:51.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:16:51.803 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:16:51 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:16:51.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:16:52.006 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:51.897+0000 7f73b22a3640 -1 osd.4 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.595531+0000 front 2024-09-17T09:16:23.595523+0000 (oldest deadline 2024-09-17T09:16:49.495412+0000) 2024-09-17T09:16:52.007 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:51 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:51.975+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:52.063 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:51 smithi079 podman[88753]: 2024-09-17 09:16:51.802037108 +0000 UTC m=+0.089541468 container create 9ff398a694615c12ba623cf08ae63c8ede700fc6a65595a13e1f70d6ee7e6c99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b) 2024-09-17T09:16:52.064 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:51 smithi079 podman[88753]: 2024-09-17 09:16:51.72848066 +0000 UTC m=+0.015984985 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:16:52.064 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:51 smithi079 podman[88753]: 2024-09-17 09:16:51.996481741 +0000 UTC m=+0.283986062 container init 9ff398a694615c12ba623cf08ae63c8ede700fc6a65595a13e1f70d6ee7e6c99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-17T09:16:52.064 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:52 smithi079 podman[88753]: 2024-09-17 09:16:52.001212659 +0000 UTC m=+0.288716979 container start 9ff398a694615c12ba623cf08ae63c8ede700fc6a65595a13e1f70d6ee7e6c99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-17T09:16:52.064 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:52 smithi079 podman[88753]: 2024-09-17 09:16:52.062785711 +0000 UTC m=+0.350290035 container attach 9ff398a694615c12ba623cf08ae63c8ede700fc6a65595a13e1f70d6ee7e6c99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-17T09:16:52.361 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:52 smithi079 podman[88753]: 2024-09-17 09:16:52.187832992 +0000 UTC m=+0.475337323 container died 9ff398a694615c12ba623cf08ae63c8ede700fc6a65595a13e1f70d6ee7e6c99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T09:16:52.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:52.089+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:52.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:52.238+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:52.701 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.3.service' 2024-09-17T09:16:52.724 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:52 smithi079 podman[88796]: 2024-09-17 09:16:52.67410755 +0000 UTC m=+0.480314287 container remove 9ff398a694615c12ba623cf08ae63c8ede700fc6a65595a13e1f70d6ee7e6c99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T09:16:52.724 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:52 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-17T09:16:52.724 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:52 smithi079 systemd[1]: Stopped Ceph osd.3 for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:16:52.724 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 17 09:16:52 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.3.service: Consumed 5.563s CPU time. 2024-09-17T09:16:52.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:16:52.739 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-17T09:16:52.739 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-17T09:16:52.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.4 2024-09-17T09:16:53.007 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:52 smithi173 systemd[1]: Stopping Ceph osd.4 for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:16:53.007 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:52.890+0000 7f73b22a3640 -1 osd.4 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.595531+0000 front 2024-09-17T09:16:23.595523+0000 (oldest deadline 2024-09-17T09:16:49.495412+0000) 2024-09-17T09:16:53.018 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:16:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:16:52.683Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.79:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.79:8765: connect: connection refused" 2024-09-17T09:16:53.018 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:16:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:16:52.683Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.79:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.79:8765: connect: connection refused" 2024-09-17T09:16:53.018 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:16:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:16:52.685Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.79:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.79:8765: connect: connection refused" 2024-09-17T09:16:53.018 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:16:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:16:52.686Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.79:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.79:8765: connect: connection refused" 2024-09-17T09:16:53.018 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:16:52 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:16:52.687Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.79:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.79:8765: connect: connection refused" 2024-09-17T09:16:53.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:53 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:53.080+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:53.507 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:53 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:53.015+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:53.507 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:53 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:53.047+0000 7f73b5c8f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:16:53.507 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:53 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:53.047+0000 7f73b5c8f640 -1 osd.4 74 *** Got signal Terminated *** 2024-09-17T09:16:53.507 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:53 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:53.047+0000 7f73b5c8f640 -1 osd.4 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:16:53.508 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:53 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:53.201+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:54.162 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:54 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:54.012+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:54.162 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:53 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:53.852+0000 7f73b22a3640 -1 osd.4 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.595531+0000 front 2024-09-17T09:16:23.595523+0000 (oldest deadline 2024-09-17T09:16:49.495412+0000) 2024-09-17T09:16:54.162 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:54 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:54.082+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:54.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:54 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:54.160+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:55.197 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:54 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:54.861+0000 7f73b22a3640 -1 osd.4 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.595531+0000 front 2024-09-17T09:16:23.595523+0000 (oldest deadline 2024-09-17T09:16:49.495412+0000) 2024-09-17T09:16:55.198 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:55 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:55.058+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:55.198 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:54 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:54.964+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:55.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:55 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:55.196+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:56.149 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:55 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:55.844+0000 7f73b22a3640 -1 osd.4 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.595531+0000 front 2024-09-17T09:16:23.595523+0000 (oldest deadline 2024-09-17T09:16:49.495412+0000) 2024-09-17T09:16:56.149 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:56 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:56.016+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:56.150 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:55 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:55.918+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:56.150 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:56 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:56.148+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:57.121 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:56 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:56.808+0000 7f73b22a3640 -1 osd.4 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.595531+0000 front 2024-09-17T09:16:23.595523+0000 (oldest deadline 2024-09-17T09:16:49.495412+0000) 2024-09-17T09:16:57.122 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:56 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:56.995+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:57.122 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:56 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:56.938+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:57.122 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:56 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:56.938+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:16:57.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:57 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:57.120+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:58.141 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:57 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4[38634]: 2024-09-17T09:16:57.824+0000 7f73b22a3640 -1 osd.4 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.595531+0000 front 2024-09-17T09:16:23.595523+0000 (oldest deadline 2024-09-17T09:16:49.495412+0000) 2024-09-17T09:16:58.141 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:58 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:58.030+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:58.141 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:57 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:57.979+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:58.141 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:57 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:57.979+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:16:58.396 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:58 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:58.139+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:58.688 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:58 smithi173 podman[73873]: 2024-09-17 09:16:58.395951199 +0000 UTC m=+5.463355789 container died 871630755c72cc5b7ba8c6d9da84a969383c70e66846a0fc8ef7d208e500ee8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True) 2024-09-17T09:16:58.990 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:58 smithi173 podman[73873]: 2024-09-17 09:16:58.794953032 +0000 UTC m=+5.862357621 container cleanup 871630755c72cc5b7ba8c6d9da84a969383c70e66846a0fc8ef7d208e500ee8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-17T09:16:58.990 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:58 smithi173 bash[73873]: ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4 2024-09-17T09:16:59.256 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:59 smithi173 podman[73895]: 2024-09-17 09:16:59.037865651 +0000 UTC m=+0.636097226 container remove 871630755c72cc5b7ba8c6d9da84a969383c70e66846a0fc8ef7d208e500ee8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:16:59.257 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:16:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:16:59.165+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:16:59.257 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:59.027+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:16:59.257 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:59.027+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:35.262069+0000 front 2024-09-17T09:16:35.261912+0000 (oldest deadline 2024-09-17T09:16:58.161907+0000) 2024-09-17T09:16:59.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:58 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:58.989+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:16:59.258 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:58 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:58.989+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:16:59.616 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:59 smithi173 podman[73989]: 2024-09-17 09:16:59.338981299 +0000 UTC m=+0.096747101 container create 6953b158e38e920a10bbda819ba5fddfa4377ccfc52654ce9b81960637fcd330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-17T09:16:59.617 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:59 smithi173 podman[73989]: 2024-09-17 09:16:59.258280668 +0000 UTC m=+0.016046436 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:16:59.617 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:59 smithi173 podman[73989]: 2024-09-17 09:16:59.516199583 +0000 UTC m=+0.273965355 container init 6953b158e38e920a10bbda819ba5fddfa4377ccfc52654ce9b81960637fcd330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-17T09:16:59.617 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:59 smithi173 podman[73989]: 2024-09-17 09:16:59.521011711 +0000 UTC m=+0.278777469 container start 6953b158e38e920a10bbda819ba5fddfa4377ccfc52654ce9b81960637fcd330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-17T09:16:59.617 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:59 smithi173 podman[73989]: 2024-09-17 09:16:59.616058075 +0000 UTC m=+0.373823847 container attach 6953b158e38e920a10bbda819ba5fddfa4377ccfc52654ce9b81960637fcd330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T09:16:59.969 INFO:journalctl@ceph.osd.4.smithi173.stdout:Sep 17 09:16:59 smithi173 podman[73989]: 2024-09-17 09:16:59.713743463 +0000 UTC m=+0.471509221 container died 6953b158e38e920a10bbda819ba5fddfa4377ccfc52654ce9b81960637fcd330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2) 2024-09-17T09:17:00.256 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:59.967+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:00.256 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:16:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:16:59.967+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:00.257 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:59.978+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:17:00.257 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:16:59 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:16:59.978+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:35.262069+0000 front 2024-09-17T09:16:35.261912+0000 (oldest deadline 2024-09-17T09:16:58.161907+0000) 2024-09-17T09:17:00.257 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:00.157+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:00.257 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:00.157+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:00.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.4.service' 2024-09-17T09:17:00.317 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:17:00.317 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-17T09:17:00.317 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-17T09:17:00.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.5 2024-09-17T09:17:00.658 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:00 smithi173 systemd[1]: Stopping Ceph osd.5 for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:00.934 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:00.657+0000 7f4638368640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:17:00.934 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:00.657+0000 7f4638368640 -1 osd.5 74 *** Got signal Terminated *** 2024-09-17T09:17:00.934 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:00.657+0000 7f4638368640 -1 osd.5 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:17:00.934 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:00.932+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:17:00.934 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:00.932+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:35.262069+0000 front 2024-09-17T09:16:35.261912+0000 (oldest deadline 2024-09-17T09:16:58.161907+0000) 2024-09-17T09:17:01.250 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:00.964+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:01.251 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:00 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:00.964+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:01.251 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:01 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:01.139+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:01.251 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:01 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:01.139+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:02.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:01 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:01.651Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:17:02.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:01 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:01.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:17:02.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:01 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:01.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:17:02.133 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:01 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:01.902+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:17:02.133 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:01 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:01.902+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:35.262069+0000 front 2024-09-17T09:16:35.261912+0000 (oldest deadline 2024-09-17T09:16:58.161907+0000) 2024-09-17T09:17:02.134 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:01 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:01.916+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:02.134 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:01 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:01.916+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:02.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:02.132+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:02.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:02.132+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:03.164 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:02.936+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:17:03.164 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:02.936+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:35.262069+0000 front 2024-09-17T09:16:35.261912+0000 (oldest deadline 2024-09-17T09:16:58.161907+0000) 2024-09-17T09:17:03.165 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:02.897+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:03.165 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:02 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:02.897+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:03.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:03 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:03.163+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:03.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:03 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:03.163+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:04.207 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:03 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:03.927+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:04.208 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:03 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:03.927+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:04.208 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:03 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:03.973+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:17:04.208 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:03 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:03.973+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:35.262069+0000 front 2024-09-17T09:16:35.261912+0000 (oldest deadline 2024-09-17T09:16:58.161907+0000) 2024-09-17T09:17:04.208 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:03 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:03.974+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:40.363296+0000 front 2024-09-17T09:16:40.363306+0000 (oldest deadline 2024-09-17T09:17:03.863244+0000) 2024-09-17T09:17:04.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:04 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:04.206+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:04.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:04 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:04.206+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:05.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:04 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:04.948+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:05.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:04 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:04.948+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:05.237 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:04 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:04.948+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:25.360514+0000 front 2024-09-17T09:16:25.360525+0000 (oldest deadline 2024-09-17T09:16:48.260406+0000) 2024-09-17T09:17:05.237 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:04 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:04.948+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:35.262069+0000 front 2024-09-17T09:16:35.261912+0000 (oldest deadline 2024-09-17T09:16:58.161907+0000) 2024-09-17T09:17:05.237 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:04 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5[44896]: 2024-09-17T09:17:04.948+0000 7f4630974640 -1 osd.5 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:40.363296+0000 front 2024-09-17T09:16:40.363306+0000 (oldest deadline 2024-09-17T09:17:03.863244+0000) 2024-09-17T09:17:05.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:05 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:05.236+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:05.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:05 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:05.236+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:06.250 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:06 smithi173 podman[74119]: 2024-09-17 09:17:06.05923585 +0000 UTC m=+5.521257870 container died 9e94094a3a02d17eba760a3d6b74b433290ae90204ebf02d67669e365a4f5e0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T09:17:06.250 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:05 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:05.929+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:06.250 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:05 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:05.929+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:06.251 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:05 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:05.929+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:06.507 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:06 smithi173 podman[74119]: 2024-09-17 09:17:06.430411305 +0000 UTC m=+5.892433323 container cleanup 9e94094a3a02d17eba760a3d6b74b433290ae90204ebf02d67669e365a4f5e0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux ) 2024-09-17T09:17:06.507 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:06 smithi173 bash[74119]: ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5 2024-09-17T09:17:06.508 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:06 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:06.249+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:06.508 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:06 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:06.249+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:06.508 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:06 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:06.249+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:41.367409+0000 front 2024-09-17T09:16:41.367384+0000 (oldest deadline 2024-09-17T09:17:06.067333+0000) 2024-09-17T09:17:07.007 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:06 smithi173 podman[74140]: 2024-09-17 09:17:06.681475489 +0000 UTC m=+0.619346277 container remove 9e94094a3a02d17eba760a3d6b74b433290ae90204ebf02d67669e365a4f5e0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:17:07.007 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:06 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:06.882+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:07.007 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:06 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:06.882+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:07.007 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:06 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:06.882+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:07.286 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:07.223+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:07.286 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:07.223+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:07.286 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:07.223+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:41.367409+0000 front 2024-09-17T09:16:41.367384+0000 (oldest deadline 2024-09-17T09:17:06.067333+0000) 2024-09-17T09:17:07.286 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:07 smithi173 podman[74232]: 2024-09-17 09:17:07.007085096 +0000 UTC m=+0.113143693 container create b08c01605e2afe8c145359e561914cfcbf89c6728c151bee6ab669b3f84a3a6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b) 2024-09-17T09:17:07.286 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:07 smithi173 podman[74232]: 2024-09-17 09:17:06.909882754 +0000 UTC m=+0.015941364 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:17:07.286 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:07 smithi173 podman[74232]: 2024-09-17 09:17:07.285349552 +0000 UTC m=+0.391408160 container init b08c01605e2afe8c145359e561914cfcbf89c6728c151bee6ab669b3f84a3a6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True) 2024-09-17T09:17:07.757 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:07 smithi173 podman[74232]: 2024-09-17 09:17:07.29013768 +0000 UTC m=+0.396196285 container start b08c01605e2afe8c145359e561914cfcbf89c6728c151bee6ab669b3f84a3a6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-17T09:17:07.757 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:07 smithi173 podman[74232]: 2024-09-17 09:17:07.3767226 +0000 UTC m=+0.482781222 container attach b08c01605e2afe8c145359e561914cfcbf89c6728c151bee6ab669b3f84a3a6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:17:07.757 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:07 smithi173 podman[74232]: 2024-09-17 09:17:07.476002084 +0000 UTC m=+0.582060698 container died b08c01605e2afe8c145359e561914cfcbf89c6728c151bee6ab669b3f84a3a6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:17:08.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.5.service' 2024-09-17T09:17:08.031 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:07.873+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:08.032 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:07.873+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:08.032 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:07 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:07.873+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:08.032 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:07 smithi173 podman[74275]: 2024-09-17 09:17:07.988148058 +0000 UTC m=+0.502580392 container remove b08c01605e2afe8c145359e561914cfcbf89c6728c151bee6ab669b3f84a3a6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:17:08.032 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:08 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.5.service: Deactivated successfully. 2024-09-17T09:17:08.032 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:08 smithi173 systemd[1]: Stopped Ceph osd.5 for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:17:08.032 INFO:journalctl@ceph.osd.5.smithi173.stdout:Sep 17 09:17:08 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.5.service: Consumed 5.198s CPU time. 2024-09-17T09:17:08.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:17:08.048 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-17T09:17:08.048 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-17T09:17:08.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.6 2024-09-17T09:17:08.377 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:08 smithi173 systemd[1]: Stopping Ceph osd.6 for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:08.377 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:08 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:08.218+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:08.377 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:08 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:08.218+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:08.377 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:08 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:08.218+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:41.367409+0000 front 2024-09-17T09:16:41.367384+0000 (oldest deadline 2024-09-17T09:17:06.067333+0000) 2024-09-17T09:17:08.377 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:08 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:08.375+0000 7f7afa66e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:17:08.377 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:08 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:08.375+0000 7f7afa66e640 -1 osd.6 74 *** Got signal Terminated *** 2024-09-17T09:17:08.378 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:08 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:08.375+0000 7f7afa66e640 -1 osd.6 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:17:09.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:08 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:08.848+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:09.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:08 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:08.848+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:09.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:08 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:08.848+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:09.757 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:09 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:09.260+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:09.757 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:09 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:09.260+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:09.757 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:09 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:09.260+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:41.367409+0000 front 2024-09-17T09:16:41.367384+0000 (oldest deadline 2024-09-17T09:17:06.067333+0000) 2024-09-17T09:17:10.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:09 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:09.822+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:10.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:09 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:09.822+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:10.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:09 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:09.822+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:10.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:10.236+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:10.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:10.236+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:10.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:10.236+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:41.367409+0000 front 2024-09-17T09:16:41.367384+0000 (oldest deadline 2024-09-17T09:17:06.067333+0000) 2024-09-17T09:17:11.238 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:10.861+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:11.238 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:10.861+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:11.238 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:10 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:10.861+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:11.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:11.237+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:11.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:11.237+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:11.508 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:11.237+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:41.367409+0000 front 2024-09-17T09:16:41.367384+0000 (oldest deadline 2024-09-17T09:17:06.067333+0000) 2024-09-17T09:17:12.023 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:11.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:17:12.023 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:11.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:17:12.023 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:11 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:11.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:17:12.251 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:11.846+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:12.251 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:11.846+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:12.251 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:11 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:11.846+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:12.506 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:12.250+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:12.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:12.250+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:12.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:12.250+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:41.367409+0000 front 2024-09-17T09:16:41.367384+0000 (oldest deadline 2024-09-17T09:17:06.067333+0000) 2024-09-17T09:17:12.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:12.250+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:46.067846+0000 front 2024-09-17T09:16:46.067835+0000 (oldest deadline 2024-09-17T09:17:11.367741+0000) 2024-09-17T09:17:13.256 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:12.824+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:13.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:12.824+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:13.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:12.824+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:13.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:12 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:12.824+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:49.643546+0000 front 2024-09-17T09:16:49.643597+0000 (oldest deadline 2024-09-17T09:17:11.943440+0000) 2024-09-17T09:17:13.757 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:13.256+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:26.464345+0000 front 2024-09-17T09:16:26.464266+0000 (oldest deadline 2024-09-17T09:16:48.764254+0000) 2024-09-17T09:17:13.757 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:13.256+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:34.465994+0000 front 2024-09-17T09:16:34.465936+0000 (oldest deadline 2024-09-17T09:16:59.765893+0000) 2024-09-17T09:17:13.757 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:13.256+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:41.367409+0000 front 2024-09-17T09:16:41.367384+0000 (oldest deadline 2024-09-17T09:17:06.067333+0000) 2024-09-17T09:17:13.757 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6[51219]: 2024-09-17T09:17:13.256+0000 7f7af6c82640 -1 osd.6 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:46.067846+0000 front 2024-09-17T09:16:46.067835+0000 (oldest deadline 2024-09-17T09:17:11.367741+0000) 2024-09-17T09:17:14.119 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:13 smithi173 podman[74360]: 2024-09-17 09:17:13.86078217 +0000 UTC m=+5.597169105 container died 3b9210d9ecbb5e4000a6e7db4d27eb97a164790856e91a3ae145baca56057e5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T09:17:14.120 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:13.835+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:14.120 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:13.835+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:14.120 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:13.835+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:14.120 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:13 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:13.835+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:49.643546+0000 front 2024-09-17T09:16:49.643597+0000 (oldest deadline 2024-09-17T09:17:11.943440+0000) 2024-09-17T09:17:14.483 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:14 smithi173 podman[74360]: 2024-09-17 09:17:14.223860712 +0000 UTC m=+5.960247644 container cleanup 3b9210d9ecbb5e4000a6e7db4d27eb97a164790856e91a3ae145baca56057e5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:17:14.483 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:14 smithi173 bash[74360]: ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6 2024-09-17T09:17:14.757 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:14 smithi173 podman[74382]: 2024-09-17 09:17:14.483058632 +0000 UTC m=+0.614819984 container remove 3b9210d9ecbb5e4000a6e7db4d27eb97a164790856e91a3ae145baca56057e5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-17T09:17:15.156 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:14.837+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:15.157 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:14.837+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:15.157 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:14.837+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:15.157 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:14 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:14.837+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:49.643546+0000 front 2024-09-17T09:16:49.643597+0000 (oldest deadline 2024-09-17T09:17:11.943440+0000) 2024-09-17T09:17:15.157 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:14 smithi173 podman[74477]: 2024-09-17 09:17:14.743147444 +0000 UTC m=+0.016260444 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:17:15.157 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:14 smithi173 podman[74477]: 2024-09-17 09:17:14.8591726 +0000 UTC m=+0.132285587 container create d148ee90aacaa9fc032dc5b127f6bc5698c8cee7a6054d2335742b8093b98e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-17T09:17:15.157 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:15 smithi173 podman[74477]: 2024-09-17 09:17:15.073054344 +0000 UTC m=+0.346167344 container init d148ee90aacaa9fc032dc5b127f6bc5698c8cee7a6054d2335742b8093b98e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0) 2024-09-17T09:17:15.157 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:15 smithi173 podman[74477]: 2024-09-17 09:17:15.077966434 +0000 UTC m=+0.351079425 container start d148ee90aacaa9fc032dc5b127f6bc5698c8cee7a6054d2335742b8093b98e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-17T09:17:15.157 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:15 smithi173 podman[74477]: 2024-09-17 09:17:15.156271088 +0000 UTC m=+0.429384091 container attach d148ee90aacaa9fc032dc5b127f6bc5698c8cee7a6054d2335742b8093b98e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux ) 2024-09-17T09:17:15.507 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:15 smithi173 podman[74477]: 2024-09-17 09:17:15.265320682 +0000 UTC m=+0.538433679 container died d148ee90aacaa9fc032dc5b127f6bc5698c8cee7a6054d2335742b8093b98e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, ceph=True) 2024-09-17T09:17:15.806 INFO:journalctl@ceph.osd.6.smithi173.stdout:Sep 17 09:17:15 smithi173 podman[74520]: 2024-09-17 09:17:15.784748433 +0000 UTC m=+0.512946803 container remove d148ee90aacaa9fc032dc5b127f6bc5698c8cee7a6054d2335742b8093b98e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=reef-19e2a2b, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:17:15.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.6.service' 2024-09-17T09:17:15.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:17:15.850 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-17T09:17:15.850 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-17T09:17:15.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.7 2024-09-17T09:17:16.174 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:15.830+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:16.174 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:15.830+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:16.174 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:15.830+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:16.174 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:15 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:15.830+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:49.643546+0000 front 2024-09-17T09:16:49.643597+0000 (oldest deadline 2024-09-17T09:17:11.943440+0000) 2024-09-17T09:17:16.174 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:15 smithi173 systemd[1]: Stopping Ceph osd.7 for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:16.506 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:16 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:16.172+0000 7f6769326640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:17:16.507 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:16 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:16.172+0000 7f6769326640 -1 osd.7 74 *** Got signal Terminated *** 2024-09-17T09:17:16.507 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:16 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:16.172+0000 7f6769326640 -1 osd.7 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:17:17.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:16 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:16.879+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:17.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:16 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:16.879+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:17.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:16 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:16.879+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:17.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:16 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:16.879+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:49.643546+0000 front 2024-09-17T09:16:49.643597+0000 (oldest deadline 2024-09-17T09:17:11.943440+0000) 2024-09-17T09:17:18.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:17 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:17.899+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:18.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:17 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:17.899+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:18.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:17 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:17.899+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:18.257 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:17 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:17.899+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:49.643546+0000 front 2024-09-17T09:16:49.643597+0000 (oldest deadline 2024-09-17T09:17:11.943440+0000) 2024-09-17T09:17:19.158 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:18 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:18.858+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:19.158 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:18 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:18.858+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:19.158 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:18 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:18.858+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:19.158 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:18 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:18.858+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:49.643546+0000 front 2024-09-17T09:16:49.643597+0000 (oldest deadline 2024-09-17T09:17:11.943440+0000) 2024-09-17T09:17:20.170 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:19 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:19.885+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:20.170 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:19 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:19.885+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:20.170 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:19 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:19.885+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:20.170 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:19 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:19.885+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:49.643546+0000 front 2024-09-17T09:16:49.643597+0000 (oldest deadline 2024-09-17T09:17:11.943440+0000) 2024-09-17T09:17:21.006 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:20 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:20.870+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6803 osd.0 since back 2024-09-17T09:16:23.240951+0000 front 2024-09-17T09:16:23.241037+0000 (oldest deadline 2024-09-17T09:16:49.140912+0000) 2024-09-17T09:17:21.006 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:20 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:20.870+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6807 osd.1 since back 2024-09-17T09:16:32.041903+0000 front 2024-09-17T09:16:32.041854+0000 (oldest deadline 2024-09-17T09:16:56.741759+0000) 2024-09-17T09:17:21.007 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:20 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:20.870+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6811 osd.2 since back 2024-09-17T09:16:42.642744+0000 front 2024-09-17T09:16:42.642725+0000 (oldest deadline 2024-09-17T09:17:05.542688+0000) 2024-09-17T09:17:21.007 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:20 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7[57614]: 2024-09-17T09:17:20.870+0000 7f6761932640 -1 osd.7 74 heartbeat_check: no reply from 172.21.15.79:6815 osd.3 since back 2024-09-17T09:16:49.643546+0000 front 2024-09-17T09:16:49.643597+0000 (oldest deadline 2024-09-17T09:17:11.943440+0000) 2024-09-17T09:17:21.757 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:21 smithi173 podman[74605]: 2024-09-17 09:17:21.58541431 +0000 UTC m=+5.510883758 container died 520b72a29a85294da62c456dddfb5d6f8481dd5208fbb369c90f60447808bab9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T09:17:22.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:21 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:21.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:17:22.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:21 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:21.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:17:22.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:21 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:21.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-09-17T09:17:22.213 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:21 smithi173 podman[74605]: 2024-09-17 09:17:21.961987337 +0000 UTC m=+5.887456829 container cleanup 520b72a29a85294da62c456dddfb5d6f8481dd5208fbb369c90f60447808bab9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-17T09:17:22.213 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:21 smithi173 bash[74605]: ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7 2024-09-17T09:17:22.506 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:22 smithi173 podman[74637]: 2024-09-17 09:17:22.212816738 +0000 UTC m=+0.625405563 container remove 520b72a29a85294da62c456dddfb5d6f8481dd5208fbb369c90f60447808bab9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T09:17:22.772 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:22 smithi173 podman[74731]: 2024-09-17 09:17:22.423637935 +0000 UTC m=+0.015730877 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T09:17:22.772 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:22 smithi173 podman[74731]: 2024-09-17 09:17:22.526934187 +0000 UTC m=+0.119027165 container create 5ef5a881c90025c965e51befd74f8236dfa93c3a3721a48b473375f2b71fdbd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:17:22.772 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:22 smithi173 podman[74731]: 2024-09-17 09:17:22.680645078 +0000 UTC m=+0.272738011 container init 5ef5a881c90025c965e51befd74f8236dfa93c3a3721a48b473375f2b71fdbd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T09:17:22.772 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:22 smithi173 podman[74731]: 2024-09-17 09:17:22.685582292 +0000 UTC m=+0.277675225 container start 5ef5a881c90025c965e51befd74f8236dfa93c3a3721a48b473375f2b71fdbd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:17:22.773 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:22 smithi173 podman[74731]: 2024-09-17 09:17:22.771473673 +0000 UTC m=+0.363566607 container attach 5ef5a881c90025c965e51befd74f8236dfa93c3a3721a48b473375f2b71fdbd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:17:23.159 INFO:journalctl@ceph.osd.7.smithi173.stdout:Sep 17 09:17:22 smithi173 podman[74731]: 2024-09-17 09:17:22.87081302 +0000 UTC m=+0.462905968 container died 5ef5a881c90025c965e51befd74f8236dfa93c3a3721a48b473375f2b71fdbd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T09:17:23.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@osd.7.service' 2024-09-17T09:17:23.481 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:17:23.481 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-17T09:17:23.481 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-17T09:17:23.481 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@rgw.foo.a 2024-09-17T09:17:23.861 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:17:23 smithi079 systemd[1]: Stopping Ceph rgw.foo.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:24.306 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:17:23 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-rgw-foo-a[69503]: 2024-09-17T09:17:23.981+0000 7f09b4229640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:17:24.306 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:17:23 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-rgw-foo-a[69503]: 2024-09-17T09:17:23.981+0000 7f09b7a94880 -1 shutting down 2024-09-17T09:17:32.111 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:31 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:31.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-17T09:17:34.272 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:17:33 smithi079 bash[88892]: time="2024-09-17T09:17:33Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-721af936-74d3-11ef-bceb-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-17T09:17:34.272 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:17:34 smithi079 podman[88892]: 2024-09-17 09:17:34.008983164 +0000 UTC m=+10.335559213 container died f48e7cfad8b622a4f6365db4c8dbdf5dd6035163543b6bb2f064684d186552c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-rgw-foo-a, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:17:34.529 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:17:34 smithi079 podman[88892]: 2024-09-17 09:17:34.377986239 +0000 UTC m=+10.704562285 container cleanup f48e7cfad8b622a4f6365db4c8dbdf5dd6035163543b6bb2f064684d186552c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-rgw-foo-a, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-17T09:17:34.529 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Sep 17 09:17:34 smithi079 bash[88892]: ceph-721af936-74d3-11ef-bceb-c7b262605968-rgw-foo-a 2024-09-17T09:17:34.754 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@rgw.foo.a.service' 2024-09-17T09:17:34.789 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:17:34.790 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-17T09:17:34.790 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-17T09:17:34.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-721af936-74d3-11ef-bceb-c7b262605968@prometheus.a 2024-09-17T09:17:35.089 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:34 smithi173 systemd[1]: Stopping Ceph prometheus.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:35.369 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:17:35.369 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T09:17:35.369 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T09:17:35.369 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:17:35.369 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T09:17:35.369 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T09:17:35.369 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T09:17:35.369 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T09:17:35.370 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T09:17:35.370 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T09:17:35.370 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T09:17:35.370 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a[68040]: ts=2024-09-17T09:17:35.088Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T09:17:35.370 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 podman[74859]: 2024-09-17 09:17:35.098772732 +0000 UTC m=+0.112712679 container died 6e0ce71149469d8f8baa9ef53c5640c1a7648a71fb59d3bcb7c8337f78e26e8b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:17:35.717 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 podman[74859]: 2024-09-17 09:17:35.466114474 +0000 UTC m=+0.480054413 container cleanup 6e0ce71149469d8f8baa9ef53c5640c1a7648a71fb59d3bcb7c8337f78e26e8b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:17:35.717 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Sep 17 09:17:35 smithi173 bash[74859]: ceph-721af936-74d3-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T09:17:35.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-721af936-74d3-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-17T09:17:35.914 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:17:35.915 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-17T09:17:35.915 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 721af936-74d3-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T09:17:36.065 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:17:37.612 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:37 smithi079 systemd[1]: Stopping Ceph alertmanager.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:38.059 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:37 smithi079 ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a[79431]: ts=2024-09-17T09:17:37.767Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:17:38.059 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:37 smithi079 podman[89354]: 2024-09-17 09:17:37.809007653 +0000 UTC m=+0.157222648 container died 21e4ae85c61a83a9799fe953541f9483885fc25f1a4393112359e05cff9c06a3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:17:38.342 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:38 smithi079 podman[89354]: 2024-09-17 09:17:38.157599568 +0000 UTC m=+0.505814550 container cleanup 21e4ae85c61a83a9799fe953541f9483885fc25f1a4393112359e05cff9c06a3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:17:38.342 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:38 smithi079 bash[89354]: ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T09:17:38.611 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:38 smithi079 podman[89370]: 2024-09-17 09:17:38.341968813 +0000 UTC m=+0.526340838 container remove 21e4ae85c61a83a9799fe953541f9483885fc25f1a4393112359e05cff9c06a3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:17:38.611 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:38 smithi079 podman[89370]: 2024-09-17 09:17:38.425916897 +0000 UTC m=+0.610288876 volume remove c6c1339e4580278ab77261fbdc8250a522c5d879b96eea658da59bbac89f8644 2024-09-17T09:17:39.112 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:17:38 smithi079 systemd[1]: Stopping Ceph node-exporter.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:39.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:38 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T09:17:39.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:38 smithi079 systemd[1]: Stopped Ceph alertmanager.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:17:39.112 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 17 09:17:38 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.299s CPU time. 2024-09-17T09:17:39.445 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:17:39 smithi079 podman[89481]: 2024-09-17 09:17:39.166593978 +0000 UTC m=+0.119905460 container died c51a61eee4484a9d5b26f34153102b912831142958269b3ca6ca52ea0d48bc92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:17:39.779 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:17:39 smithi079 podman[89481]: 2024-09-17 09:17:39.545072304 +0000 UTC m=+0.498383819 container cleanup c51a61eee4484a9d5b26f34153102b912831142958269b3ca6ca52ea0d48bc92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:17:39.779 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:17:39 smithi079 bash[89481]: ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a 2024-09-17T09:17:40.042 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:17:39 smithi079 podman[89501]: 2024-09-17 09:17:39.779158012 +0000 UTC m=+0.614593084 container remove c51a61eee4484a9d5b26f34153102b912831142958269b3ca6ca52ea0d48bc92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:17:40.042 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:17:39 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:17:40.043 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:17:39 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T09:17:40.043 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:17:39 smithi079 systemd[1]: Stopped Ceph node-exporter.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:17:40.043 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 17 09:17:39 smithi079 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 2.553s CPU time. 2024-09-17T09:17:40.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 721af936-74d3-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T09:17:40.657 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:17:42.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:41 smithi173 systemd[1]: Stopping Ceph iscsi.iscsi.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:42.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:42 smithi173 ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a[62653]: debug Shutdown received 2024-09-17T09:17:52.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:52 smithi173 bash[75247]: time="2024-09-17T09:17:52Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-17T09:17:52.338 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:52 smithi173 podman[75247]: 2024-09-17 09:17:52.055368051 +0000 UTC m=+10.127559094 container died 14190907a849c76640fbea0a70a299c57c25cb49cfcf61d627fb454b259cf4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T09:17:52.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:52 smithi173 podman[75247]: 2024-09-17 09:17:52.427723055 +0000 UTC m=+10.499914091 container cleanup 14190907a849c76640fbea0a70a299c57c25cb49cfcf61d627fb454b259cf4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b) 2024-09-17T09:17:52.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:52 smithi173 bash[75247]: ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a 2024-09-17T09:17:53.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:52 smithi173 podman[75269]: 2024-09-17 09:17:52.653723228 +0000 UTC m=+0.597549765 container remove 14190907a849c76640fbea0a70a299c57c25cb49cfcf61d627fb454b259cf4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-17T09:17:53.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:52 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-17T09:17:53.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:52 smithi173 podman[75318]: 2024-09-17 09:17:52.877163701 +0000 UTC m=+0.106213836 container died 1eb3c04b85bd24023e6aa01baabd80c27d3488148fd1a5c59ea1598d6b1a1963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux ) 2024-09-17T09:17:53.449 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:53 smithi173 podman[75318]: 2024-09-17 09:17:53.230939085 +0000 UTC m=+0.459989215 container cleanup 1eb3c04b85bd24023e6aa01baabd80c27d3488148fd1a5c59ea1598d6b1a1963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-17T09:17:53.449 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:53 smithi173 bash[75318]: ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu 2024-09-17T09:17:53.449 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:53 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-17T09:17:53.449 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:53 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Unit process 62506 (conmon) remains running after unit stopped. 2024-09-17T09:17:53.449 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:53 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Unit process 75334 (podman) remains running after unit stopped. 2024-09-17T09:17:53.450 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:53 smithi173 systemd[1]: Stopped Ceph iscsi.iscsi.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:17:53.450 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:53 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Consumed 2.700s CPU time. 2024-09-17T09:17:53.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:17:53 smithi173 systemd[1]: Stopping Ceph grafana.a for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:53.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Sep 17 09:17:53 smithi173 podman[75334]: 2024-09-17 09:17:53.457111119 +0000 UTC m=+0.583555357 container remove 1eb3c04b85bd24023e6aa01baabd80c27d3488148fd1a5c59ea1598d6b1a1963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-17T09:17:54.144 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:17:53 smithi173 podman[75460]: 2024-09-17 09:17:53.848046237 +0000 UTC m=+0.093486188 container died 8606b326879cff927394a6fa75e6e749bb60d37e2593635f13a5404adc0b1fb6 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a, io.openshift.expose-services=, io.openshift.tags=base rhel8, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, version=8.5) 2024-09-17T09:17:54.445 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:17:54 smithi173 podman[75460]: 2024-09-17 09:17:54.218711883 +0000 UTC m=+0.464151834 container cleanup 8606b326879cff927394a6fa75e6e749bb60d37e2593635f13a5404adc0b1fb6 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a, version=8.5, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, vendor=Red Hat, Inc., io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , architecture=x86_64, vcs-type=git, distribution-scope=public, com.redhat.component=ubi8-container, description=Ceph Grafana Container) 2024-09-17T09:17:54.446 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:17:54 smithi173 bash[75460]: ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a 2024-09-17T09:17:54.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:17:54 smithi173 podman[75476]: 2024-09-17 09:17:54.444781525 +0000 UTC m=+0.599056239 container remove 8606b326879cff927394a6fa75e6e749bb60d37e2593635f13a5404adc0b1fb6 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.component=ubi8-container, vcs-type=git, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, name=ubi8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, description=Ceph Grafana Container, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-09-17T09:17:54.757 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:17:54 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:17:54.758 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:17:54 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-17T09:17:54.758 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:17:54 smithi173 systemd[1]: Stopped Ceph grafana.a for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:17:54.758 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Sep 17 09:17:54 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@grafana.a.service: Consumed 6.860s CPU time. 2024-09-17T09:17:55.062 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:17:54 smithi173 systemd[1]: Stopping Ceph node-exporter.b for 721af936-74d3-11ef-bceb-c7b262605968... 2024-09-17T09:17:55.315 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:17:55 smithi173 podman[75597]: 2024-09-17 09:17:55.061319071 +0000 UTC m=+0.116388055 container died eb913e3b3a96f8cb6aeeaef1ccf8a45f048c64564a7b22847ce21616cd1ed59e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:17:55.674 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:17:55 smithi173 podman[75597]: 2024-09-17 09:17:55.414846883 +0000 UTC m=+0.469915857 container cleanup eb913e3b3a96f8cb6aeeaef1ccf8a45f048c64564a7b22847ce21616cd1ed59e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:17:55.675 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:17:55 smithi173 bash[75597]: ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b 2024-09-17T09:17:55.983 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:17:55 smithi173 podman[75614]: 2024-09-17 09:17:55.674265568 +0000 UTC m=+0.616736810 container remove eb913e3b3a96f8cb6aeeaef1ccf8a45f048c64564a7b22847ce21616cd1ed59e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-721af936-74d3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:17:55.984 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:17:55 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:17:55.984 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:17:55 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T09:17:55.984 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:17:55 smithi173 systemd[1]: Stopped Ceph node-exporter.b for 721af936-74d3-11ef-bceb-c7b262605968. 2024-09-17T09:17:55.984 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Sep 17 09:17:55 smithi173 systemd[1]: ceph-721af936-74d3-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 2.570s CPU time. 2024-09-17T09:17:56.472 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:17:56.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:17:56.532 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-17T09:17:56.534 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908264/remote/smithi079/crash 2024-09-17T09:17:56.535 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/crash -- . 2024-09-17T09:17:56.590 INFO:teuthology.orchestra.run.smithi079.stderr:tar: /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T09:17:56.590 INFO:teuthology.orchestra.run.smithi079.stderr:tar: Error is not recoverable: exiting now 2024-09-17T09:17:56.592 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908264/remote/smithi173/crash 2024-09-17T09:17:56.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/crash -- . 2024-09-17T09:17:56.669 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/721af936-74d3-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T09:17:56.670 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-09-17T09:17:56.671 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-17T09:17:56.671 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/721af936-74d3-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-09-17T09:17:56.701 INFO:teuthology.orchestra.run.smithi079.stderr:grep: /var/log/ceph/721af936-74d3-11ef-bceb-c7b262605968/ceph.log: No such file or directory 2024-09-17T09:17:56.703 INFO:tasks.cephadm:Compressing logs... 2024-09-17T09:17:56.703 DEBUG:teuthology.orchestra.run.smithi079:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T09:17:56.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T09:17:56.771 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T09:17:56.772 INFO:teuthology.orchestra.run.smithi079.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T09:17:59.397 INFO:tasks.cephadm:Archiving logs... 2024-09-17T09:17:59.398 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908264/remote/smithi079/log 2024-09-17T09:17:59.399 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T09:17:59.857 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908264/remote/smithi173/log 2024-09-17T09:17:59.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T09:18:00.163 INFO:tasks.cephadm:Removing cluster... 2024-09-17T09:18:00.163 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 721af936-74d3-11ef-bceb-c7b262605968 --force 2024-09-17T09:18:00.303 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:18:00.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 721af936-74d3-11ef-bceb-c7b262605968 --force 2024-09-17T09:18:00.724 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 721af936-74d3-11ef-bceb-c7b262605968 2024-09-17T09:18:00.987 INFO:tasks.cephadm:Removing cephadm ... 2024-09-17T09:18:00.987 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T09:18:01.004 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T09:18:01.020 INFO:tasks.cephadm:Teardown complete 2024-09-17T09:18:01.021 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T09:18:01.030 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T09:18:01.030 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T09:18:01.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T09:18:01.133 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T09:18:01.145 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T09:18:01.145 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T09:18:01.148 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T09:18:01.160 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-09-17T09:18:01.162 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-09-17T09:18:01.164 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T09:18:01.164 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-09-17T09:18:01.164 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 33 -174us[ -167us] +/- 44ms 2024-09-17T09:18:01.164 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -1207us[-1199us] +/- 49ms 2024-09-17T09:18:01.164 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +2054us[+2062us] +/- 69ms 2024-09-17T09:18:01.164 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:18:01.166 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T09:18:01.166 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-09-17T09:18:01.166 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 29 -92us[ -92us] +/- 44ms 2024-09-17T09:18:01.166 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -1120us[-1118us] +/- 49ms 2024-09-17T09:18:01.166 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +2135us[+2137us] +/- 69ms 2024-09-17T09:18:01.166 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:18:01.167 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T09:18:01.178 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T09:18:01.178 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T09:18:01.192 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T09:18:01.283 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T09:18:01.356 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T09:18:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:18:01.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T09:18:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:18:01.414 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T09:18:01.428 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T09:18:01.442 INFO:teuthology.task.internal:Duration was 1365.386434 seconds 2024-09-17T09:18:01.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T09:18:01.457 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T09:18:01.457 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T09:18:01.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T09:18:01.496 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T09:18:01.498 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T09:18:01.899 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T09:18:01.899 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-09-17T09:18:01.899 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T09:18:01.959 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T09:18:01.959 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T09:18:01.983 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T09:18:01.983 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T09:18:02.002 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T09:18:02.052 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T09:18:02.052 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T09:18:02.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T09:18:02.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T09:18:02.356 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T09:18:02.356 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T09:18:02.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T09:18:02.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T09:18:02.418 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T09:18:02.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T09:18:02.444 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-09-17T09:18:02.469 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-09-17T09:18:02.498 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T09:18:02.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:18:02.523 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T09:18:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:18:02.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T09:18:02.566 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T09:18:02.567 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908264/remote/smithi079 2024-09-17T09:18:02.568 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T09:18:02.609 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908264/remote/smithi173 2024-09-17T09:18:02.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T09:18:02.646 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T09:18:02.646 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T09:18:02.649 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T09:18:02.695 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T09:18:02.709 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T09:18:02.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T09:18:02.724 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T09:18:02.725 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T09:18:02.727 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T09:18:02.740 INFO:teuthology.orchestra.run.smithi079.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 09:18 /home/ubuntu/cephtest 2024-09-17T09:18:02.750 INFO:teuthology.orchestra.run.smithi173.stdout: 265762 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 09:18 /home/ubuntu/cephtest 2024-09-17T09:18:02.751 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T09:18:02.764 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T09:18:02.796 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} duration: 1365.3864336013794 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-17T09:18:02.797 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T09:18:02.864 INFO:teuthology.run:pass